Zoom control device, control method of zoom control device, and recording medium

ABSTRACT

A subject detection unit of an imaging apparatus detects a subject image from an image. An automatic zoom control unit performs zoom control according to a size of a subject detected by the subject detection unit. The automatic zoom control unit automatically selects a specific composition among a plurality of compositions and sets a reference size of the subject used to control a zoom magnification based on the selected composition and the size and position of the detected subject. A process of determining a scene using information including a detection result by the subject detection unit, a composition selection process is performed on the determination scene, and one composition is selected from a composition of upper body of the subject, a composition of whole body, a composition of a subject face, and a composition of multiple people.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technology for detecting a subjectimage from an image and controlling zoom magnification.

2. Description of the Related Art

In the related art, there are imaging apparatuses that have a functionof driving a zoom lens and performing optical magnification (hereinafterreferred to as optical zoom) and a function of expanding a part of aphotographed image and performing electronic magnification (hereinafterreferred to as electronic zoom). The imaging apparatuses also have anautomatic zoom function (automatic magnification) of automaticallychanging a zoom magnification according to detection information ofsubjects. Hereinafter, an optical zoom operation and an electronic zoomoperation are collectively referred to as a zoom operation.

Japanese Patent Laid-Open No. 2012-95019 discloses the configuration ofa camera equipped with an automatic zoom function of constantlymaintaining the size of a subject. Japanese Patent Laid-Open No.2012-95019 discloses a method in which a photographer selects acomposition among a plurality of options, “face-up,” “bust-up,” and“whole body,” in an automatic zoom function. In automatic zoom controlof maintaining the size of a subject image, the size of the subjectimage serving as a basis of zoom control is decided based on a selectedcomposition. For example, when a photographer selects the composition“whole body,” a zoom operation is performed so that an image of thewhole body of a person that is a subject fits in a photographing screen.

In Japanese Patent Laid-Open No. 2012-95019, however, it is necessary toselect a composition desired by a photographer in advance in anautomatic zoom function. Therefore, whenever a photographing scene or asubject state (a motion, a number, or the like) is changed, it isnecessary for the photographer to perform a setting changingmanipulation. When setting of the composition is not appropriate for aphotographing scene, there is a possibility of a subject not beingimaged with an appropriate size. For example, when a composition of“face” is set in a scene in which a moving subject or a distant subjectis photographed, there is a possibility of a subject falling outside ofa frame. When a composition of “whole body” is set in a scene in which astopped subject is photographed, there is a possibility of a subjectbeing considerably small in a composition.

When a specific subject is desired to be photographed with the samecomposition, it is necessary to perform a setting changing manipulationagain whenever a subject is changed. For example, when a baby is desiredto be photographed with a close-up on his or her “face” or a child isdesired to be photographed with the composition of “whole body,” thephotographer has to manually change setting of the composition when asubject changes.

SUMMARY OF THE INVENTION

The present invention enables automatic zoom control to be performedwith a composition suitable for a scene determined based on detectioninformation or authentication information of a subject or an attributeof the determined subject without composition selection of aphotographer.

According to an aspect of the present invention, there is provided azoom control device comprising: a subject detection unit configured todetect a subject image from an image; a composition selection unitconfigured to automatically select one composition among a plurality ofcompositions based on information regarding the subject image detectedby the subject detection unit; a setting unit configured to set areference size of the subject image used to control a zoom magnificationbased on the composition selected by the composition selection unit anda size and a position of the subject image detected by the subjectdetection unit; and a control unit configured to control the zoommagnification based on the reference size set by the setting unit andthe size of the subject image detected sequentially by the subjectdetection unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof a digital camera according to an embodiment of the present invention.

FIG. 2 is a diagram exemplifying a relation between a focal distance andthe position of a focus lens for each subject distance.

FIG. 3 is explanatory diagrams illustrating a process of preventing asubject (object) from falling outside of a screen.

FIGS. 4A to 4C are explanatory diagrams illustrating a process ofpreventing a subject (person) from falling outside of a screen.

FIGS. 5A and 5B are explanatory diagrams illustrating a process ofpreventing a change in a size of a subject (person) inside a screen.

FIGS. 6A to 6E are diagrams for describing composition setting when asubject is a person.

FIG. 7 is a flowchart for describing the flow of a process for anautomatic zoom function.

FIG. 8 is a flowchart for describing a subject searching process.

FIGS. 9A and 9B are diagrams for describing a search end region of thesubject searching process.

FIGS. 10A and 10B are flowcharts for describing a subject designationprocess.

FIG. 11 is a flowchart for describing another example of the subjectdesignation process.

FIG. 12 is a flowchart for describing a reference size setting process.

FIG. 13 is a flowchart for describing a screen size calculation process.

FIG. 14 is a transition diagram for describing automatic compositiondetermination in composition setting “auto.”

FIGS. 15A to 15C are diagrams illustrating subject positions and screensizes in the outermost periphery.

FIG. 16 is a flowchart for describing a process of calculating thereference size in the composition setting “auto.”

FIG. 17 is a flowchart for describing a process of calculating a subjectposition ratio in a horizontal direction.

FIG. 18 is a flowchart for describing a process of calculating a subjectposition ratio in a vertical direction.

FIG. 19 is a table illustrating the number of faces for estimating thesubject position.

FIG. 20 is a flowchart for describing an automatic zoom control process.

FIG. 21 is a flowchart for describing a zoom operation.

FIG. 22 is a transition diagram for describing automatic compositiondetermination in the composition setting “auto” according to a secondembodiment.

FIG. 23 is a table illustrating face ratios according to age at the timeof subject authentication.

FIG. 24 is a transition diagram for describing automatic compositiondetermination in the composition setting “auto” at the time of thesubject authentication.

FIG. 25 is a table illustrating an initial composition and a transitioncondition of the composition at the time of the subject authentication.

FIG. 26 is a table illustrating the number of faces for which a bodyposition is estimated at the time of the subject authentication.

FIG. 27 is a table illustrating composition selection for each kind ofsubject at the time of the subject authentication.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described indetail according to the appended drawings. FIG. 1 is a block diagramillustrating an example of the configuration of a digital camera 100that has an automatic zoom function and is an example of an imagingdevice including a zoom control device according to the embodiment.

A lens group is held in a lens barrel 101. A zoom lens 102 is moved inan optical axis direction of the lens barrel 101 to adjust a focaldistance and optically change an angle of view (move a zoom position). Afocus lens 103 is moved in the optical axis direction of the lens barrel101 to perform focusing. A vibration-proof lens (image blur correctionlens) 104 corrects image blur caused due to camera shake or the like. Anaperture and shutter 105 that adjust an amount of light are used forexposure control. In the embodiment, the digital camera 100 is animaging apparatus in which the lens barrel 101 and a camera body areintegrated, but the invention is not limited thereto. The embodiment canalso be applied to an imaging system configured to include a camera bodyand an interchangeable lens detachably mounted on the camera body.

An image sensor 106 generates an imaging signal by receiving lightpassing through the lens barrel 101 and converting a subject image intoan electric signal through photoelectric conversion. The image sensor106 is, for example, a charge-coupled device (CCD) type or acomplementary metal oxide semiconductor (CMOS) type image sensor. Theimaging signal generated by the image sensor 106 is input to an imageprocessing circuit 107 and is subjected to various processes such as apixel interpolation process and a color conversion process. Image dataafter the various processes is stored in an image memory 108. The imagememory 108 is a storage device such as a dynamic random access memory(DRAM) or a static random access memory (SRAM).

The display unit 109 is configured to include a thin film transistor(TFT) drive type liquid crystal display (LCD) and displays aphotographed image (image data) or specific information (for example,photographed information). An electronic viewfinder (EVF) function bywhich a photographer performs alignment of an angle of view can beprovided through information display such as live view related to aphotographed image.

An aperture and shutter driving unit 110 calculates exposure controlvalues (an aperture value and a shutter speed) based on luminanceinformation obtained through image processing in the image processingcircuit 107 and drives the aperture and shutter 105 based on acalculation result. In this way, automatic exposure (AE) control isperformed. A vibration-proof lens driving unit 111 calculates adeflection amount applied to the digital camera 100 based on deflectiondetection information by an angular velocity sensor such as a gyrosensor. Depending on the calculation result, the vibration-proof lens104 is driven so that the deflection amount applied to the digitalcamera 100 is cancelled (reduced).

A focus lens driving unit 112 drives the focus lens 103. In theembodiment, the digital camera 100 performs automatic focusing (AF)control according to a contrast scheme. That is, based on focusinginformation (contrast evaluation value) of a photographing opticalsystem obtained through image processing in the image processing circuit107, the focus lens driving unit 112 drives the focus lens 103 so that asubject comes into focus. However, the invention is not limited thereto,but a phase difference AF scheme may be used as the AF control insteadof the contrast scheme. Further, the AF control may be performedaccording to a plurality of schemes such as a combination of thecontrast scheme and another scheme.

A zoom lens driving unit 113 drives the zoom lens 102 according to azoom manipulation instruction. A manipulation unit 117 includes a zoomlever or a zoom button as a zoom manipulation member used for thephotographer to instruct the camera to perform zooming. A system controlunit 114 performs control of detecting a manipulation amount and amanipulation direction of the zoom manipulation member used for a zoominstruction manipulation, calculating a zoom driving speed or a zoomdriving direction, and moving the zoom lens 102 in an optical axisaccording to a calculation result.

Image data generated through a photographing operation is transmitted toa recording unit 116 via an interface (I/F) unit 115 to be stored. Theimage data is recorded in one or both of an external recording mediumand a nonvolatile memory 118. The external recording medium is, forexample, a memory card that is mounted on the digital camera 100 foruse. The nonvolatile memory 118 is a storage medium embedded in thedigital camera 100. The memory 118 stores not only program data or imagedata but also setting information of the digital camera 100 orinformation regarding a zoom-in position or the like in an automaticzoom function to be described below.

A manipulation unit 117 includes a release switch for instructingphotographing start and an automatic zoom manipulation switch forinstructing start or end of the automatic zoom function in addition tothe foregoing zoom manipulation member. A signal from the manipulationunit 117 is transmitted to the system control unit 114.

The system control unit 114 includes a calculation device such as acentral processing unit (CPU). The system control unit 114 controls theentire digital camera 100 by transmitting a control command to each unitin response to a manipulation of the photographer. The system controlunit 114 executes various control programs stored in the memory 118, forexample, programs for performing control of the image sensor 106, AE/AFcontrol, zoom control (including an automatic zoom process), and thelike.

To maintain a focused state when an angle of view is changed throughoptical zoom, it is necessary to move the focus lens 103 to anappropriate focus position according to the position of the zoom lens102 when the lens barrel 101 is of a rear focus type. Such control isreferred to as computer zoom (CZ) control and is performed by a CZcontrol unit 119 in the system control unit 114. FIG. 2 is a diagramexemplifying a relation between a focal distance of the zoom lens and afocus position for each subject distance. In FIG. 2, the relationbetween the focal distance of the zoom lens and the focused focusposition is graphed as a data table indicated for each distance up to asubject. In the embodiment, the table is referred to as a focus camtable. In FIG. 2, the horizontal axis represents the focal distancecorresponding to the zoom position and the vertical axis represents thefocus position. A distance (subject distance) from the digital camera100 to a subject is exemplified on the side of each graph line.

The system control unit 114 performs a scan operation by controlling thefocus lens driving unit 112 and moving the focus lens 103 within apredetermined range when the AF control is performed. A focus positionwhich is a focused point is detected using a contrast evaluation valueor the like obtained during a scan operation according to a knownmethod. A subject distance is measured using the zoom position and thefocus position at that time with reference to the focus cam table.

Next, control related to the automatic zoom function in the systemcontrol unit 114 will be described. As illustrated in FIG. 1, the systemcontrol unit 114 includes the CZ control unit 119, an electronic zoomcontrol unit 120, an automatic zoom control unit (hereinafter referredto as an AZ control unit) 122, and a subject detection unit 123.

The digital camera 100 has an optical zoom function and an electroniczoom function, and the CZ control unit 119 and the zoom lens drivingunit 113 serve to control the optical zoom. The CZ control unit 119detects a zoom position of the zoom lens 102 for each predeterminedcontrol period at the time of a zoom operation. Then, the CZ controlunit 119 performs control of driving the focus lens 103 to track thefocus cam table at a subject distance measured in the AF controlaccording to the detected zoom position. In this way, it is possible toperform the optical zoom operation while maintaining the focused state.

On the other hand, the electronic zoom control unit 120 and the imagememory 108 serve to control the electronic zoom. The electronic zoomcontrol unit 120 realizes the electronic zoom function by clipping datain a target region from image data transmitted to the image memory 108.The electronic zoom control unit 120 realizes smooth electronic zoomdisplay by causing the display unit 109 to perform display whilegradually increasing a range clipped at a frame rate period of an imagecaptured in the image sensor 106.

The subject detection unit 123 detects a desired subject region fromimage data stored in the image memory 108. In the embodiment, a subjectdetection method (a face detection process or a color detection process)of detecting a subject (the face of a person or the like or an object)based on face information or color information included in the imagedata will be described.

The face detection process is a process of detecting a face regionpresent in image data by a known algorithm. For example, the subjectdetection unit 123 extracts a feature amount from a rectangular partialregion on the image data and compares the feature amount to a featureamount of a face prepared in advance. Then, the subject detection unit123 determines that the partial region is a face region when acorrelation value of both features exceeds a predetermined thresholdvalue. By repeating this determination process while changing acombination of a size, a disposition position, and a disposition angleof the partial region, it is possible to detect various face regionspresent in the image data.

In the color detection process, a process of storing color informationof a subject region designated according to a subject designation methodto be described below as a feature color is performed. The colordetection process is performed when a detection target subject is anobject (“object” other than a person). For example, an RGB signal or aluminance signal (Y signal) which are output signals from the imageprocessing circuit 107 and a color difference (R-Y or B-Y) signal areused as the color information. When a subject is detected, the subjectdetection unit 123 divides image data into a plurality of partialregions and calculates an average value of luminance and colordifferences for each partial region. The subject detection unit 123compares feature color information stored in advance to colorinformation of each region at the time of the detection of the subjectand sets a partial region in which a difference between the luminanceand the color difference is equal to or less than a predetermined amountas a candidate for a subject region. A process of setting a group ofpartial regions mutually adjacent in the candidate of the subject regionas a same-color region and setting a region in which the same-colorregion is within a predetermined size range as a final subject region isperformed.

In the embodiment, a case in which the subject detection unit 123 has afunction of registering information regarding a subject and a functionof authenticating the subject will be described, but a registration unitand an authentication unit may be independently provided. That is, theregistration unit performs processes of acquiring image information ofthe subject and registering the image information in association withattribute information of the subject in a storage unit. The subjectauthentication unit performs correlation calculation of correlating afeature amount of the detected subject image and a feature amount of thesubject image obtained from the image information of the subjectregistered in advance and performs an authentication process for thesubject based on a correlation value.

Next, an authentication method (a face authentication process and anobject authentication process) of determining an attribute of thedetected subject in the subject detection unit 123 will be described. Inthe face authentication process, a process of registering a face imageof a person which is a photographing target and attributes (a name, anage, a date of birth, a sex, an initial composition, and the like) ofthe face in advance is performed. A pattern matching process isperformed on all of the registered face images and the face imagesdetected through the face detection process by a known algorithm. Aprocess of authenticating the registered face image with the highestcorrelation value as a detected face is performed. When the correlationvalues of all the registered face images are less than a predeterminedvalue, the detected face is determined to be an unregistered face.Accordingly, it is possible to specify the attributes of the person whois the photographing target. On the other hand, in the objectauthentication process, a process of storing an image of an object otherthan the person who is the photographing target as a template image inthe memory 118 is performed. For example, there is a method of storingrepresentative subjects such as airplanes, electric trains, birds, andthe moon as template images in advance or a method of registering imagesphotographed by the photographer as template images. To improvedetection performance, a process of storing images of the same object atdifferent angles or with different colors as a plurality of pieces ofimage data is performed. Further, a process of registering attributeinformation (kind, determination information whether an object is movingor stationary, a setting composition, and the like) of an objectcorresponding to the template image is performed. In addition to colorinformation of an object region detected in the color detection process,pattern matching is performed on shape information, texture information,or the like with the registered template images. A process ofauthenticating an image with the highest correlation value as a detectedobject is performed. When correlation values of all the template imagesare less than a predetermined value, a detected object region isdetermined to be an unregistered object. Accordingly, it is possible tospecify the attributes of the object which is a photographing target.

The subject detection unit 123 calculates the size of the subject regionin the image data using subject distance information measured by the CZcontrol unit 119 and focal distance information of the zoom lens 102. Aposture detection unit 124 detects a posture (for example, normalposition/grip-up/grip-down) of the digital camera 100 based oninformation of an acceleration sensor. A shake detection unit 125detects a deflection state of the digital camera 100 based on angularvelocity information or the like by a gyro sensor by determination. Theshake detection unit 125 determines that the camera is handheld when adeflection amount (detected value) applied to the gyro sensor or thelike is equal to or greater than a predetermined amount (thresholdvalue), and determines that the camera is fixed on a tripod or the likewhen the deflection amount is less than the predetermined amount.Sensors of a detection unit that acquires control information of thevibration-proof lens driving unit 111 may also be configured to be usedas the acceleration sensor and the gyro sensor used to detect theposture and detect the shake.

Next, an overview of the automatic zoom function and the AZ control unit122 according to the embodiment will be described. In a camera notequipped with the automatic zoom function, the following manipulation isnecessary, for example, when a subject moves and falls outside of theframe while the photographer performs framing in a telephoto state andwaits to take a picture.

First, the photographer performs a zoom-out operation by manipulatingthe zoom manipulation member, and then searches for a subject. After thephotographer searches for the subject, the photographer performs a zoommanipulation again to adjust an angle of view until the angle of theview becomes a desired angle of view. For example, when the size of asubject image changes due to movement of the subject, the photographernecessarily manipulates the zoom manipulation member to adjust the sizeof the subject image.

Conversely, in a camera equipped with the automatic zoom function, thephotographer performs a manipulation of designating a subject on a touchpanel or the like to designate a subject desired to be photographedafter setting the automatic zoom function. When the automatic zoomfunction is set, a zoom operation is automatically performed so that thedesignated subject has a predetermined size near the center of an image.As the method of designating a subject, in addition to the touch panelmanipulation, for example, there is a method of designating a subjectnear the center of a screen when the photographer manipulates a specificbutton or a method of automatically selecting a main subject amongsubjects detected by a camera.

The subject detection unit 123 calculates the position or size of asubject region designated from the image memory 108 in image data. Amotion of a subject can be tracked by performing this process on sampledimage data continuously whenever an image is displayed as a live view.The AZ control unit 122 starts the zoom-out operation when an image of asubject being tracked is detected in a zoom-out region to be describedbelow or when an image of the detected subject becomes greater than apredetermined size. That is, the AZ control unit 122 instructs the CZcontrol unit 119 or the electronic zoom control unit 120 to performzoom-out in a wide angle direction. When the subject is detected in azoom-in region and the subject image fits within a range of apredetermined size, a zoom-in operation is performed to the telephotoside. Through such a process, the photographer may operate the camerawithout worrying about the zoom operation so that an image of a desiredsubject fits in a screen. Even when the subject falls outside of theframe, a zoom magnification is automatically changed. Therefore, it ispossible to perform alignment of an angle of view more simply.

Next, start conditions of a zoom-out operation or a zoom-in operationwill be described with reference to FIGS. 3 to 5B. FIG. 3 is explanatorydiagrams illustrating a process of preventing a subject (object) fromfalling outside of a screen. FIGS. 4A to 4C are explanatory diagramsillustrating a process of preventing a subject (person) from fallingoutside of a screen.

In FIGS. 3 to 4C, frames 300 a and 300 b are first tracking frames(hereinafter referred to as object tracking frames) for tracking asubject (object) and frames 400 a to 400 f are second tracking frames(hereinafter referred to as face tracking frames) for tracking a subject(the face of a person). Hereinafter, for descriptions that could applyto a person or an object, the object tracking frame and the facetracking frames are collectively referred to as a subject tracking framein some cases. The subject tracking frame is displayed to surround asubject image on an electronic viewfinder screen (EVF screen) of thedisplay unit 109 so that the photographer can see a designated subject.The position and size of the subject tracking frame on the screen iscalculated based on the face information and the color information bythe subject detection unit 123 and is updated at a frame rate period.

A process of preventing a subject (airplane) from falling outside of ascreen will be described with reference to FIG. 3. A zoom-out region ZO(hereinafter referred to as a ZO region) in a left figure of FIG. 3indicates a region more outside than a predetermined ratio in the entireangle of view (entire screen) displayed by the EVF. For example, thecentral point of the screen is 0% and the entire screen is 100%, and acase in which a position located at 80% of the entire screen is set as aboundary of the ZO region is assumed. In this case, a region of 80% to100% of the entire screen is the ZO region. When a part of the objecttracking frame 300 a enters the ZO region, the AZ control unit 122performs control such that a zoom-out operation starts. The AZ controlunit 122 stores a zoom magnification (corresponding to a zoom-in angleof view) before zoom movement in the memory 118. A target zoommagnification or a zoom speed during a zoom-out operation are set inadvance according to the size of a subject image or a movement speed.The target zoom magnification or the zoom speed may be appropriatelycalculated according to the size of a subject image or the movementspeed. The zoom-out operation is performed according to the target zoommagnification or the zoom speed. In this way, it is possible toefficiently prevent a subject from falling outside of the frame.

A right figure of FIG. 3 illustrates an angle of view when a zoom-outoperation corresponding to a predetermined zoom change amount isperformed from the angle of view in the left figure of FIG. 3. A zoom-inregion ZI (hereinafter referred to as a ZI region) illustrated in theright figure of FIG. 3 indicates a region more inside than apredetermined ratio at a zoom-in angle of view (an angle of view beforezoom-out) 301 in a subject search state. For example, the central pointof the screen is 0% and the zoom-in angle of view 301 is 100%, and acase in which a position located at 70% of the zoom-in angle of view 301is set as a boundary of the ZI region is assumed. In this case, a regionof 0% to 70% of the entire zoom-in angle of view 301 is the ZI region.At this time, for example, when a zoom-out magnification is ½, thezoom-in angle of view 301 has a size of 50% of the entire screen.Accordingly, the boundary of the ZI region is 70%×(½)=35% of the entirescreen. The ZI region can also be said to be a region of 0% to 35% ofthe entire screen. When the photographer changes a direction of thecamera so that the object tracking frame 300 b fits in the ZI region,the AZ control unit 122 performs control such that a zoom-in operationstarts.

Next, a process of preventing a subject (person) from falling outside ofa screen will be described with reference to FIGS. 4A to 4C. When asubject is a person and a part of the face tracking frame enters the ZOregion, a zoom-out operation starts. When the face tracking frame fitsin the ZI region, a zoom-in operation is performed. The case in which asubject is a person is different from the case in which a subject is anobject. Thus, since a movement direction of the subject can be predictedto some extent, the ZO region and the ZI region are set according to aregion in the predicted movement direction. In photographing in thehandheld state, a subject may fall outside of the frame due to aninfluence of camera shake or the like. However, when the subject fallsoutside of the frame due to camera shake or the like, the photographercan include the subject in the frame again by performing an operation ofincluding the subject in the frame. Here, when a ZO region is set in anupper portion of the screen and a person is disposed near the center toperform photographing, the face tracking frame enters the ZO region andthe subject may fall outside of the frame unintentionally. Accordingly,when a subject is a person in the handheld state (a state in which thephotographer holds the camera), no ZO region is set in an upper portionof the screen in consideration of a framing operation of thephotographer.

In this way, in the embodiment, when the subject detection unit 123detects a face, the AZ control unit 122 changes the ZO region and the ZIregion according to a posture of the camera or a shake state. Theposture of the camera is detected by the posture detection unit 124 andthe shake state is determined from a detection result obtained by theshake detection unit 125. The detection result obtained by the shakedetection unit 125 refers to a detection result indicating whether thecamera is in the handheld state. Hereinafter, this will be describedspecifically with reference to FIGS. 4A to 4C.

FIG. 4A illustrates a ZO region and a ZI region set when thephotographer holds the camera and poses at a normal position. In such aphotographing scene, when a subject moves in a horizontal direction andfalls outside of the frame, the position of the subject in a screenmoves in the horizontal direction (longitudinal direction) of the screenat the normal position. Thus, both of the ZO region and the ZI regionare arranged in a vertical belt-like shape in the vertical direction(transverse direction) with respect to the screen at the normalposition. The ZO region is set in the vertical belt-like shape locatedon both end sides in the horizontal direction in a horizontally longrectangular screen. The ZI region is set in the vertical belt-like shapelocated in the central portion of the screen. In this case, when theface tracking frame 400 a enters the ZO region, the AZ control unit 122determines that zoom-out starts and performs control of the zoom-outoperation corresponding to a predetermined zoom magnification. When theface tracking frame 400 b is included in the ZI region, the AZ controlunit 122 determines that zoom-in starts and performs control of thezoom-in operation corresponding to a predetermined zoom magnification upto a zoom return position. By setting the ZO region and the ZI region inthis way, it is possible to efficiently prevent the subject (person)from falling outside of the frame.

FIG. 4B illustrates the ZO region and the ZI region set when the postureof the camera is changed in the same photographing scene and thephotographer poses the camera in a vertical position state of grip-downor grip-up. In this case, the ZO region and the ZI region are arrangedin vertical belt-like shapes in the vertical direction (longitudinaldirection) of a screen at a vertical position. That is, the ZO region isset in the vertical belt-like shape located on both end sides in thehorizontal direction in a vertically long rectangular screen. The ZIregion is set in the vertical belt-like shape located in the centralportion of the screen. In this case, when the face tracking frame 400 centers the ZO region, the AZ control unit 122 determines that zoom-outstarts and performs control of the zoom-out operation corresponding to apredetermined zoom magnification. When the face tracking frame 400 d isincluded in the ZI region, the AZ control unit 122 determines thatzoom-in starts and performs control of the zoom-in operationcorresponding to a predetermined zoom magnification up to a zoom returnposition. By setting the ZO region and the ZI region in this way, it ispossible to detect the motion of the subject in the horizontal directionand efficiently prevent the subject (person) from falling outside of theframe.

FIG. 4C illustrates the ZO region and the ZI region set when a detectionstate of the shake detection unit 125 is a fixed state. The fixed staterefers to a state in which the camera is fixed on a tripod or the like,and there is no possibility of a subject falling outside of the framedue to camera shake. When a zoom-in operation is performed while asubject is not framed near the center of a screen, there is apossibility of the subject falling outside of the frame due to thezoom-in operation. Accordingly, the ZO region is set in the entireperiphery portion of the screen and the ZI region is set further insidethan a zoom-in angle of view. That is, the ZO region is set in arectangular belt-like shape located on the corner sides in the verticaland horizontal directions of the screen. The ZI region is set in arectangular shape located in the center of the screen. In this case,when a face tracking frame 400 e enters the ZO region, the AZ controlunit 122 determines that zoom-out starts and performs control of thezoom-output operation corresponding to a predetermined zoommagnification. When a face tracking frame 400 f is included in the ZIregion, the AZ control unit 122 determines that zoom-in starts andperforms control of the zoom-in operation corresponding to apredetermined zoom magnification up to a zoom return position.

In the embodiment, as described above, the range of each of the ZOregion and the ZI region is dynamically changed according to a change inthe posture of the camera or the photographing state (handheldstate/fixed state). In this way, it is possible to efficiently prevent asubject from falling outside of a frame while preventing an erroneousoperation due to camera shake or the like. Further, the ZO region or theZI region may be changed according to one of the posture of the cameraand the photographing state (handheld state/fixed state) or only one ofthe ZO region and the ZI region may be changed.

Next, a zoom operation of maintaining a ratio of a subject image to ascreen within a predetermined range will be described. In theembodiment, control (size maintenance control) is performed so that thesize of the subject image is maintained within the predetermined rangefrom a reference size by automatically performing a zoom operation whenthe size of the detected subject image exceeds a predetermined multipleof the reference size. FIGS. 5A and 5B are diagrams for describing aprocess of maintaining the size of a subject (person) image in a screen.FIGS. 6A to 6E are explanatory diagrams of composition setting of asubject (person).

First, setting of a range (composition) in which a tracking targetsubject fits in a screen will be described with reference to FIGS. 6A to6E. FIG. 6A exemplifies screen display when composition setting issetting of “manual.” In the “manual” setting, the photographer changesthe size of a face tracked when the photographer performs a manual zoomoperation through a zoom lever manipulation while viewing an image of aperson on a screen. The size of the subject image at that time is storedas a reference size in the memory 118. FIG. 6B exemplifies screendisplay when the composition setting is setting of “face.” In thecomposition setting of “face,” a size with which a face fits in thescreen in accordance with a posture of the camera or a direction of theface is calculated as a reference size and is stored in the memory 118.Similarly, FIG. 6C exemplifies screen display when the compositionsetting is setting of “upper body” and FIG. 6D exemplifies screendisplay when the composition setting is setting of “whole body.” Areference size is calculated so that the reference size is a size set oneach screen and is stored in the memory 118.

FIG. 6E exemplifies screen display when the composition setting issetting of “auto.” In the “auto” setting, the AZ control unit 122determines an appropriate composition based on a subject position, asubject size, the number of subjects, a posture of the camera, or thelike in the screen. The reference size is calculated so that thedetermined composition is realized and is stored in the memory 118. Amethod of calculating the reference size will be described below withreference to FIG. 12.

The photographer can perform a manipulation of changing the compositionsetting by manipulating left and right buttons of the manipulation unit117 on a photographing screen or performing selection on a setting menu.When the photographer performs the manipulation of changing thecomposition setting, the AZ control unit 122 updates informationregarding the composition setting stored in the memory 118. FIGS. 6A to6E illustrate the composition setting examples of “manual,” “face,”“upper body,” and “auto” when the subject is a person, but the settingof the composition is not limited thereto. Only some of the fiveexemplified compositions may be set or other compositions may beincluded. When a subject is an object, for example, compositions of“manual,” “large,” “medium,” “small,” and “auto” may be set.

A case in which the composition setting is “manual” will be described asan example with reference to FIGS. 5A and 5B in the zoom operation ofmaintaining the ratio of the subject image to the screen within thepredetermined range. FIG. 5A exemplifies a zoom operation performedautomatically by the camera when a subject (person) approaches thecamera. This is a zoom-out operation of fitting the ratio of the subjectimage to the screen within the predetermined ratio. In FIGS. 5A and 5B,face tracking frames 500 a to 500 f are displayed to surround a faceregion as a feature region of the person who is the subject.Accordingly, the size of the face tracking frame will be described as asubject size herein.

A left figure of FIG. 5A illustrates an angle of view when a subject isdesignated according to a subject designation method to be describedbelow. The size of the face tracking frame 500 a at the time of thedesignation of the subject is stored as a reference subject size(reference size) in the memory 118. A center figure of FIG. 5Aillustrates an angle of view when the subject approaches the camera whenthe zoom magnification is not changed from the state of the left drawingin the left figure of FIG. 5A. For example, a size which is 150% of thesize of the face tracking frame 500 a which is the reference subjectsize is set as a start size of the zoom-out operation. When a relationbetween the subject tracking frames (face tracking frames) is “facetracking frame 500 b>face tracking frame 500 a×150%,” that is, thetracking frame is changed with respect to the reference size more than apredetermined change amount, the AZ control unit 122 determines that thezoom-out operation starts.

A right figure of FIG. 5A shows an angle of view zoomed out from theangle of view 501 in the center figure of FIG. 5A by a predeterminedzoom magnification and the face tracking frame 500 c. Here, thepredetermined zoom magnification is set to 1/1.5 in consideration of achange ratio (150%) of the size of the face tracking frame at the timeof the start of the zoom-out operation with respect to the referencesubject size. Thereafter, when the subject further approaches thecamera, the zoom-out to the wide angle side is further performed so thatthe subject image can continuously fit within the predetermined ratio.Accordingly, the photographer can concentrate on manipulating therelease switch.

In contrast, FIG. 5B exemplifies a zoom operation performedautomatically by the camera when the person who is a subject moves awayfrom the camera. This is a zoom-in operation of fitting a ratio of thesubject image to the screen within a predetermined range. A left figureof FIG. 5B shows an angle of view when a subject is designated accordingto a subject designation method to be described below. The size of theface tracking frame 500 d at the time of the designation of the subjectis stored as a reference subject size in the memory 118 (when thecomposition setting is “manual”).

A center figure of FIG. 5B shows an angle of view when the subject movesaway from the camera when the zoom magnification is not changed from thestate of the left figure of FIG. 5B. For example, a size which is 50% ofthe size of the face tracking frame 500 d which is the reference subjectsize is set as a start size of the zoom-in operation. When a relationbetween the face tracking frames is “face tracking frame 500 e>facetracking frame 500 d×50%,” the tracking frame is determined to bechanged with respect to the reference size more than a predeterminedchange amount. When the determination condition is satisfied and theface tracking frame 500 e is included in the ZI region, the AZ controlunit 122 determines that the zoom-in operation starts. Here, the ZIregion is set further inside than an angle of view 502 zoomed in by apredetermined zoom magnification with respect to the angle of view ofthe center figure of FIG. 5B.

A right figure of FIG. 5B shows an angle of view (corresponding to theangle of view 502) zoomed in from the angle of view in the center figureof FIG. 5B by the predetermined zoom magnification and the face trackingframe 500 f. Here, the predetermined zoom magnification is set to 1/0.5in consideration of a change ratio (50%) of the size of the facetracking frame at the time of the start of the zoom-in operation withrespect to the reference subject size.

The processes of preventing a subject from falling outside of a framehave been described when the subject is an object in FIG. 3 and when thesubject is a person in FIGS. 4A to 4C. The processes of fitting theratio of the size of the subject image to the screen within thepredetermined ranges have been described when the subject is a person inFIGS. 5A and 5B. Even when a tracking target subject is an object, zoomoperation start determination may be performed to control maintenance ofthe size of the subject image illustrated in FIGS. 5A and 5B as in thecase in which the subject is a person. When the composition setting is asetting other than “manual,” a zoom operation is automatically performedaccording to each reference size as in FIGS. 5A and 5B.

Next, a process for the automatic zoom function will be described withreference to FIGS. 7 to 27. FIG. 7 is a flowchart illustrating an entireprocessing example of the automatic zoom function. Hereinafter, theautomatic zoom function is assumed to be performed based on aninstruction of the system control unit 114 (the AZ control unit 122)unless explicitly mentioned otherwise.

In S700, the AZ control unit 122 first determines a manipulation stateof the automatic zoom manipulation switch of the manipulation unit 117.When the automatic zoom manipulation switch is pressed, the processproceeds to S701. When the automatic zoom manipulation switch is notpressed, the determination process of S700 is repeated. In S701, asubject searching process is performed. A subject designation process ofS702, a reference size setting process of S703, and automatic zoomcontrol of S704 are further performed in order.

Immediately after the process of the automatic zoom function starts orwhen a subject search button of the manipulation unit 117 is pressed(yes in S705), the subject searching process of S701 is performed. Thesubject searching process performed in step S701 in control of a zoommagnification will be described with reference to FIG. 8. When aphotographing target subject is far away from the camera, even though asubject image is in the screen, there is a possibility of the subjectbeing undetectable by the subject detection unit 123 when a featureregion of the subject is too small. In such a scene, the automatic zoomfunction based on subject information may not be performed. Accordingly,a subject searching process of performing a zoom-in operation whiledetecting a subject up to a predetermined focal distance is performed.

In S800, a process of determining whether the subject search button ispressed immediately after start of automatic zoom or during theautomatic zoom rather than immediately after the start of the automaticzoom is performed. In the case of the former positive determinationresult (yes) in S800, the process proceeds to S801. In the case of thenegative determination result (no), the process proceeds to S803. InS801, a process of acquiring a subject search end focal distance isperformed. The subject search end focal distance is a focal distance atan end zoom magnification when the searching of the subject image in theimage ends. Hereinafter, the subject search end focal distance isreferred to as a search end focal distance. In the embodiment, thesearch end focal distance is set to a focal distance of 85 mm by 35 mmsize conversion immediately after the start of the automatic zoom. Thefocal distance is a focal distance of a standard lens generallyconsidered to be appropriate when a person is photographed. Even whenthere is no subject (person) or a face is undetectable (for example, aperson is facing backward), the subject detection can be set to astandby state at an appropriate angle of view after the end of thesearching. Here, the focal distance is not limited to 85 mm. Anotherfocal distance (for example, 50 mm or 100 mm) corresponding to apredetermined (fixed) zoom magnification may be used irrespective of thezoom magnification at a current time. The photographer may change thesearch end focal distance from the menu setting or can change the searchend focal distance according to a photographing mode, a subject to bephotographed, or the like. When the process proceeds from S801 to S802,the focal distance at the time of mode start is compared to the searchend focal distance and it is determined whether the focal distance is ona wide angle side or a telephoto side from the search end focaldistance. When the focal distance at the time of the start of theautomatic zoom is on the telephoto side from the search end focaldistance (no in S802), the subject searching process ends. Conversely,when the focal distance at the time of the start of the automatic zoomis on the wide angle side from the search end focal distance (yes inS802), the process proceeds to S805.

Conversely, when it is determined in S800 that the subject is searchedfor through a manipulation of the subject search button, a process ofacquiring the search end focal distance is performed in S803. In theembodiment, the search end focal distance in the manipulation of thesubject search button is assumed to be a focal distance on the telephotoside 2.5 times the focal distance when the manipulation is performed.Here, this magnification is merely an example, and another magnificationmay be used. The photographer may change the search end focal distancefrom the menu setting or can change the search end focal distanceaccording to a photographing mode, a subject to be photographed, or thelike. The process proceeds from S803 to S804, the search end focaldistance acquired in S803 is compared to the focal distance at a teleend (telephoto end), and it is determined whether the search end focaldistance is on the wide angle side or the telephoto side from the focaldistance at the tele end. When the search end focal distance is on thetelephoto side from the focal distance at the tele end, the subjectsearching process ends. When the search end focal distance is on thewide angle side from the focal distance at the tele end, the processproceeds to S805. In S805, the AZ control unit 122 sets a zoom changeamount corresponding to the search end focal distance acquired in S801or S803 in the CZ control unit 119 or the electronic zoom control unit120 and instructs to perform the zoom-in operation.

Next, in S806, it is determined whether the current focal distancereaches the search end focal distance. When the current focal distancereaches the search end focal distance, the process proceeds to S809.After the zoom-in operation stops, the subject searching process ends.Conversely, when it is determined in S806 that the current focaldistance does not reach the search end focal distance, the processproceeds to S807. S807 is a process of determining whether a subject isdetected by the subject detection unit 123. When it is determined inS807 that the subject is detected, the process proceeds to S808. When itis determined that no subject is detected, the process returns to S806.In S808, it is determined whether a detection position of the subject iswithin a predetermined search end region of the screen. Thepredetermined search end region is assumed to be decided in accordancewith a threshold value (position determination reference value) set inadvance according to a subject. When the detection position of thesubject detected in S807 is within the predetermined search end regionon the screen, the process proceeds to S809, the zoom-in operationstops, and the subject searching process ends. Conversely, when it isdetermined in S808 that the detection position of the subject is notwithin the predetermined search end region on the screen, the processreturns to S806 and the determination processes from S806 to S808continue.

Here, the predetermined search end region described in S808 will bedescribed with reference to FIGS. 9A and 9B. When a specific subject (aperson or an object) is photographed, the subject which is aphotographing target is disposed near the center of the screen in somecases. When a subject which is not the photographing target is detectedin the periphery of the screen and the subject searching process ends,there is a possibility of the subject searching process ending beforethe subject which is the photographing target is detected. In this case,the automatic zoom function may not be performed according to thephotographer's intention. Accordingly, in the embodiment, only when asubject is detected in a central region of the screen which is aspecific region and within a region near this region, is the search endregion set as a region for determination so that the subject searchingprocess ends. In FIG. 9A, a search end region 900 a set when the subjectis an object (airplane) is exemplified. When the subject is an object, arange of a predetermined ratio, for example, 70%, from the center of thescreen in any of upward, downward, right, and left directions on thescreen is set as the search end region of the subject. In FIG. 9B, onthe other hand, a search end region 900 b set when the subject is aperson is exemplified. When the subject is a person, the search endregion is set so that a range of 100% from the center of the screen in aregion of an upper portion of the screen is included and a range of apredetermined ratio, for example, 70%, from the center of the screen inanother direction (right, left, or down) is included. When the subjectis a person, there is a possibility of a face tracking frame 902 of asubject which is a photographing target being disposed in the upperportion of the screen. For this reason, even when a subject is detectedin the upper portion of the screen, the search end region is set so thatthe subject searching process ends.

Next, the subject designation process indicated in step S702 in FIG. 7will be described with reference to FIGS. 10A and 10B. FIG. 10A is aflowchart illustrating an example of a process in which a userdesignates a desired subject using the touch panel included in themanipulation unit 117. In this example, the user is assumed to perform adesignating manipulation by touching a subject image displayed on thedisplay unit 109. In S1000, the AZ control unit 122 determines whetherthe touch panel is pressed. When the touch panel is pressed, the processproceeds to S1001. When the touch panel is not pressed, thedetermination process of S1000 is repeated.

In S1001, the AZ control unit 122 acquires information regarding aposition (touch position) touched by the user. Subsequently, in S1002,the AZ control unit 122 notifies the subject detection unit 123 of thetouch position and the subject detection unit 123 performs facedetection near the touch position. When the face is detected near thetouch position in S1002, a main subject is determined to be a person andthe process proceeds to S1003. Conversely, when the face is not detectednear the touch position in S1002, the main subject is determined to bean object other than a person and the process proceeds to S1004.

In S1003, the AZ control unit 122 performs control such that faceinformation of the person who is an automatic tracking target is storedin the memory 118. As the specific face information, there isinformation regarding a size of the face, a detection position of theface, a direction of the face, or the like when the subject isdesignated. In a camera having a face authentication function,identification information such as an authentication ID is also storedin the memory 118. In S1004, the AZ control unit 122 performs controlsuch that a feature color near the touch position is stored as colorinformation of the automatic tracking target in the memory 118. As thespecific color information, there is information regarding a featurecolor and its luminance, a chromatic value or a size of a same-colorregion, the central position of the same-color region, or the like atthe time of the subject designation. In the camera having the objectauthentication function, identification information such as anauthentication ID is also stored in the memory 118. In the followingdescription, the face information and the color information arecollectively referred to as subject information (including a subjectsize and a subject detection position).

After the subject information is stored in the memory 118 in S1003 orS1004, the process proceeds to S1005 and the subject tracking frame isdisplayed. In S1005, the AZ control unit 122 performs control such thatthe subject detection position is set as a center and a subject trackingframe (an object tracking frame or a face tracking frame) with a sizecorresponding to the subject size is displayed on the display unit 109.Thereafter, the subject designation process ends.

In this way, the subject detection unit 123 in the system control unit114 detects a subject at a position designated on the display unit 109by the photographer or near the designated position. Then, the AZcontrol unit 122 causes the display unit 109 to display the subjecttracking frame. According to the example of the process of FIG. 10A, thesubject that the photographer desires to track can be designated simplyaccording to an intuitive method. However, the subject designationmethod is not limited to this method. Another method will be describedwith reference to FIG. 10B.

FIG. 10B is a flowchart illustrating an example of a process when theuser designates a subject using a different switch (subject designationswitch) from the automatic manipulation switch as a manipulation memberincluded in the manipulation unit 117. In S1006, the AZ control unit 122first displays a frame serving as a reference of the subject destinationnear the center of the screen of the display unit 109. The photographeradjusts the direction of the camera so that an image of a subject thathe or she desires to track using the frame as the reference is fittednear the center of the screen. Next, in S1007, the AZ control unit 122determines whether the subject designation switch is pressed. When thesubject designation switch is pressed, the process proceeds to S1008.When the subject destination switch is not pressed, the determinationprocess of S1007 is repeated in a standby state.

In S1008, the subject detection unit 123 performs the face detectionnear the center of the screen. When the face is detected near the centerof the screen, a main subject is determined to be a person and theprocess proceeds to S1009. Conversely, when the face is not detectednear the center of the screen, the main subject is determined to be anobject other than a person and the process proceeds to step S1010. Sincethe same process as S1003 of FIG. 10A is performed in S1009 and the sameprocess as S1004 of FIG. 10A is performed in S1010, the descriptionthereof will be omitted. After the subject information is stored in thememory 118 in S1009 or S1010, the process proceeds to S1011. Then, thesubject tracking frame (the object tracking frame or the face trackingframe) is displayed and the subject designation process ends.

In this way, in the processing example of FIG. 10B, the subjectdetection unit 123 detects the subject at the position of the center ofthe screen or near the position of the center of the screen of thedisplay unit 109. The AZ control unit 122 causes the display unit 109 todisplay the subject tracking frame indicating the position of thesubject. Even in a camera on which a manipulation member such as a touchpanel is not mounted, the user can simply designate a subject.

FIG. 11 is a flowchart illustrating an example of a process ofautomatically selecting a subject to be tracked among faces detectedwhen the automatic zoom manipulation switch included in the manipulationunit 117 is pressed. In S1100, the subject detection unit 123 firstperforms the face detection on the entire screen to determine whetherthe face of a person is detected. When the face of only one person isdetected on the entire screen, a main subject is determined to be aperson and the process proceeds to step S1101. Conversely, when one faceis not detected in the screen, the process proceeds to S1105.

In S1101, when the number of people whose faces are detected is oneperson, the subject detection unit 123 selects the face as a main face.When a plurality of faces are detected, the subject detection unit 123selects a main face assumed as a subject to be tracked among the faces.As the determination standard for selecting the main face, for example,there is a method of preferentially selecting a face of which a facedetection position is located closer to the vicinity of the center ofthe screen. When a plurality of faces are at substantially the sameposition, there is a method of selecting a face with a larger size as amain face. When a camera has a face authentication function, there is amethod of preferentially setting a face registered for authentication asa main face when the face registered for authentication is present.

The process proceeds to S1101 to S1102. Then, after the AZ control unit122 stores the face information regarding the main face in the memory118, the process proceeds to S1103 and the face tracking frame isdisplayed on the screen. Next, in S1104, the face changing determinationis performed. That is, when the plurality of faces are detected, theautomatically selected main face is not necessarily the face intended bythe photographer. For this reason, the photographer can change the mainface. At this time, when the photographer presses a switch (theautomatic zoom manipulation switch or the like) of the manipulation unit117 and designates a desired face, a process of changing the main faceto another face that was not selected as the main face among the facesfor which the face tracking frame was detected is performed. When themain face is changed in S1104, the process returns to S1102 and the faceinformation to be stored in the memory 118 is updated. In S1103, theface tracking frame is changed to the size and the detection position ofthe newly selected main face.

Conversely, the process proceeds from S1100 to S1105 when the mainsubject is determined to be an object other than a person. Then, the AZcontrol unit 122 performs control such that a feature color near thecenter of the screen is stored as color information of the automatictracking target in the memory 118. Next, in S1106, the subjectdesignation process ends after the object tracking frame is displayed onthe screen. The basic processes in S1102, S1105, S1103 (S1106) of FIG.11 are respectively the same as the processes in S1003, S1004, and 1005of FIG. 10A.

In this way, in the processing example of FIG. 11, the subject detectionunit 123 in the system control unit 114 performs the face detection onthe entire screen of the display unit 109. When a plurality of faces aredetected, the AZ control unit 122 performs control such that the displayunit 109 displays a subject tracking frame indicating the position of afirst face as a main subject among the plurality of faces. When thesubject is changed from the first face to a second face, the AZ controlunit 122 performs control such that the display unit 109 displays asubject tracking frame indicating the position of the second face. Thephotographer can designate the desired subject simply with fewermanipulations.

In the subject designation process, both of the method of FIG. 10A andthe method of FIG. 10B may be simultaneously applied when a cameraincludes both of the touch panel and the subject designation switchincluded in the manipulation unit 117. After the subject change isdetermined in S706 illustrated in FIG. 7, the subject can be designatedwith the method of FIG. 10A or 10B. After the subject searching processof S701, the subject can also be designated with the method of FIG. 11.The subject designation process applied according to a flow of theprocess may be changed. When the subject designation process (S702 ofFIG. 7) ends, the process proceeds to S703 of FIG. 7 to perform aprocess of setting the size of the tracked subject as a reference size.

The normal reference size setting process of S703 illustrated in FIG. 7will be described with reference to FIGS. 12 to 19. Further, a referencesize setting process when the subject registered in advance is detectedand authenticated through the subject authentication process will bedescribed with reference to FIGS. 23 to 27. FIG. 12 is a flowchartillustrating an example of the reference size setting process when asubject is a person. In S1200, the AZ control unit 122 first determineswhether the composition setting stored in the memory 118 is “manual”described above. When the composition setting is “manual,” the processproceeds to S1201. When the composition setting is not “manual,” theprocess proceeds to S1203.

In S1201, the AZ control unit 122 performs subject detectiondetermination. When the subject detection unit 123 detects that thesubject is a person, the AZ control unit 122 determines whether the faceis detected. When the subject is an object, the AZ control unit 122determines whether the same feature color is detected. The determinationprocess of S1201 is repeated until the subject is detected. When thesubject is detected, the process proceeds to S1202. In S1202, the AZcontrol unit 122 sets the size of the detected subject image as areference size and stores the size in the memory 118. Then, the processends. Accordingly, when the composition setting is “manual,” the size(detection size) of the subject image first detected at the time ofstarting of the reference size setting process is stored as thereference size in the memory 118.

In S1203, the AZ control unit 122 calculates a screen size based on theposture information of the camera determined by the posture detectionunit 124. This screen size is used to calculate a reference size inS1211, S1704, S1705, S1806, and S1807 to be described below. A screensize calculation process will be described with reference to FIG. 13. Inthe embodiment, a case in which the screen size of a still image is setto “long sidexshort side=640×480” and the screen size of a moving imageis set to “long sidexshort side=640×360” will be exemplified. However,the screen size is not limited thereto.

In S1300, the posture detection unit 124 determines whether thephotographer holds the camera at the normal position or holds the cameraat the vertical position. When it is determined that the photographerholds the camera at the normal position, the process proceeds to S1301.When the photographer holds the camera at the vertical position, theprocess proceeds to S1306. In S1301, a process of setting 640 which is along side size to the screen size in the horizontal direction isperformed. Next, in S1302 a process of determining whether a current setmode is a mode in which a still image and a moving video can besimultaneously photographed (still image/moving image simultaneousphotographing mode) is performed. The still image/moving imagesimultaneous photographing mode is a mode in which the photographerpresses the release switch of the manipulation unit 117 duringphotographing of a moving image to simultaneously perform photographingof a still image or a form of a framing manipulation immediately beforephotographing of a still image is automatically recorded as a movingimage. When it is determined in S1302 that the current set mode is thestill image/moving image simultaneous photographing mode, a process ofcomparing the short side size of the moving image to the short side sizeof the still image and setting the smaller size as a vertical screensize is performed. This is because a subject is prevented from fallingoutside of a screen at any angle of view of the still image and themoving image by performing automatic zoom in accordance with the smallscreen size. Accordingly, in the embodiment, when the current set modeis the still image/moving image simultaneous photographing mode, theprocess proceeds to S1304 to set 360 which is a short side size of themoving image as the vertical screen size. Thereafter, the screen sizecalculation process ends.

When it is determined in S1302 that the current set mode is not theimage/moving image simultaneous photographing mode, the process proceedsto S1303. In S1303, it is determined whether a moving image is beingrecorded or a still image is being framed. When it is determined thatthe moving image is being recorded, the process proceeds to S1304.Conversely, when it is determined that the moving image is not beingrecorded, the process proceeds to S1305. In S1305, 480 which is a shortside size of the still image is set as the vertical screen size.

When it is determined in S1300 that the photographer holds the camera atthe vertical position, the process proceeds from S1306 to S1310. Theprocesses from S1306 to S1310 are processes in which the setting of thehorizontal screen size and the vertical screen size is switched in theprocesses from S1301 to S1305. That is, since “horizontal” and“vertical” can be replaced with each other in the above description, thedetailed description will be omitted. In the embodiment, the example inwhich the screen size of a still image is 4:3 (see FIGS. 15A and 15B)and the screen size of a moving image is 16:9 (see FIG. 15C) isdescribed, but the screen size may be changed according to an aspectratio. A ratio of an image of a person to a screen can be set to beconstant irrespective of the aspect ratio. The example in which thescreen size differs between the still image and the moving image in theimage/moving image simultaneous photographing mode has been described,but a plurality of still images or moving images with different screensizes may also be photographed simultaneously.

When the screen size is decided in S1203 of FIG. 12, the processproceeds to S1204. When the composition setting stored in the memory 118is one of “face,” “upper body,” and “whole body,” the AZ control unit122 decides a ratio (referred to as a face ratio) of the size of a faceimage to the screen according to the composition setting in S1204 toS1209. The face ratio according to the embodiment is a ratio of thelength of the face tracking frame to the screen in the verticaldirection, but may be a ratio of the length of the face tracking frameto the screen in the horizontal direction, an area ratio, or the like.Specifically, for example, it is determined in S1204 whether thecomposition setting is “face.” When the composition setting is “face”(yes in S1204), the face ratio is decided as 27% (S1207). When thecomposition setting is not “face,” the process proceeds to S1205 todetermine whether the composition setting is “upper body.” When thecomposition setting is “upper body” (yes in S1205), the face ratio isdecided as 16% (S1208). When the composition setting is not “upperbody,” the process proceeds to S1206 to determine whether thecomposition setting is “whole body.” When the composition setting is“whole body” (yes in S1206), the face ratio is decided as 7% (S1209).After the face ratio is decided in the steps from S1204 to S1209, theprocess proceeds to S1210.

In S1210, the AZ control unit 122 calculates the reference size bymultiplying the face ratio decided in S1204 to S1209 by the verticalscreen size calculated in S1203. This is a method of calculating thereference size from the face ratio to the screen size in the verticaldirection on the presupposition that the subject (person) is standing.When the direction of the face can be determined by the subjectdetection unit 123, the direction of the screen size may be decidedbased on the vertical direction of the face on the screen. Further,after the process of storing the calculated reference size in the memory118 is performed, the reference size setting process ends. Accordingly,when the composition setting is one of “face,” “upper body,” and “wholebody,” a size with a predetermined face ratio according to thecomposition setting to the screen size is set as the reference size.

When the detected subject is a face registered in advance, a referencesize more appropriate for the subject can be calculated by changing aface ratio according to the age of the subject. FIG. 22 illustrates faceratios set according to an age of the subject authenticated through theface authentication process. For the age of the subject, there is amethod of using registered age information without change as theattribute of the subject. Further, when the date of birth is registered,the present age of the subject can be calculated based on the date ofbirth and the present date. In the embodiment, when the detected subjectis not authenticated (unregistered face), the same face ratio as a ratiowhen the subject is a 2-year-old to 12-year-old child is set. When thecomposition setting is “face,” the face ratio is set to 27% irrespectiveof the age of the subject. When the composition setting is one of “upperbody” or “whole body,” the face ratio is changed according to the age ofthe subject. For example, when the age of the subject is a 0-year-old to1-year-old baby, the face ratio is set to a value greater than a valuein the case of the non-authentication. When the subject is an adult (interms of body shape) of 13 years or older, the face ratio is set to avalue less than a value in the case of the non-authentication.Accordingly, when the composition setting is “upper body” or “wholebody,” an appropriate size can be obtained so that the image size of thesubject is not considerably larger or considerably smaller than thescreen size.

When the composition setting is not “whole body” in S1206, that is, thecomposition setting is determined to be “auto,” the process proceeds toS1211. Then, the AZ control unit 122 automatically determines anappropriate composition according to a photographing state, the numberof subjects, a motion of the subject, or the like. An automaticcomposition determination and composition selection process in thecomposition setting of “auto” of S1211 will be described with referenceto FIGS. 14 and 24. The composition setting “auto” is a mode in whichcomposition matching can be performed more simply by determining aphotographing scene and causing the camera to automatically select anappropriate composition (automatic composition selection).

FIG. 14 is a transition diagram of the composition automaticallyselected in the embodiment. In the embodiment, a composition selectionprocess of selecting an appropriate composition among five compositions,upper body 1400, whole body 1401, face 1402, multiple people 1403, andmanual 1404, according to a photographing scene is performed. In acomposition matching manipulation, it is necessary to set a subjectimage such that the subject image has an appropriate size instantlybefore photographing when a still image is photographed. On the otherhand, when a moving image is photographed, it is necessary to set asubject image such that the subject image normally has an appropriatesize during the photographing such as continuous tracking of a subjector approach of a stopped subject. When a moving image is photographed,image data at the time of a zoom-in operation or a zoom-out operationfrom a change in the composition is recorded. Therefore, a video can beeffectively photographed by changing the composition at an appropriatetiming. In this way, an appropriate composition differs between a stillimage and a moving image. Accordingly, a process of changing theselected composition in accordance with a photographing state isperformed during still image framing 1405 and during moving imagerecording 1406. That is, in the embodiment, a photographing preparationstate of a still image is exemplified as a first operation state and arecording state of a moving image is exemplified as a second operationstate. One composition is selected as the composition of a selectioncandidate belonging to a first composition group during the still imageframing 1405 from upper body 1400, whole body 1401, and multiple people1403. One composition is selected as the composition of a selectioncandidate belonging to a second composition group during the movingimage recording 1406 from upper body 1400, whole body 1401, and face1402. That is, the compositions of upper body 1400 and whole body 1401are common to the first and second composition groups, the compositionof multiple people 1403 is included in only the first composition group,and the composition of face 1402 is included in only the secondcomposition group. According to the specification of the imagingapparatus or the like, three or more operation states and a plurality ofcomposition groups (including a composition selectable in each operationstate) corresponding to the operation states can be set.

Here, determination conditions of photographing scenes and transition ofeach composition will be described. When the automatic compositiondetermination process S1211 of the composition setting “auto” starts,upper body 1400 is set as the composition of an initial state. Thecomposition of upper body 1400 is a composition selected when a stoppedsingular subject is determined to be a photographing scene during thestill image framing 1405 or during the moving image recording 1406. Insuch a scene, a composition of approach to a subject can beappropriately set by setting a composition in which the upper body ofthe subject is fitted in the screen.

When a motion of a photographing target subject is detected in thecomposition of upper body 1400 or when a distance between the camera andthe subject is determined to be greater than a predetermined distance D1(first threshold value), the composition is changed to whole body 1401.The composition of whole body 1401 is a composition selected when ascene in which a moving subject or a distant subject, that is, a subjectthat could easily fall outside of a screen, is photographed isdetermined during the still image framing 1405 or during the movingimage recording 1406. In such a scene, by tracking the subject in thecomposition in which the whole body of the subject is fitted, thesubject is set to fall outside of the screen as little as possible. Whenit is detected that the photographing target subject stops for apredetermined time or more in the composition of whole body 1401 and itis determined that the distance between the camera and the subject isless than a predetermined distance D2 (second threshold value), aprocess of returning the composition to upper body 1400 is performed.The predetermined time is a determination standard time set in advance.

Hereinafter, a method of determining a motion or stop of the subjectwill be described. When at least one of a change amount of position orsize of the subject detected by the subject detection unit 123 or ashake amount detected by the shake detection unit 125 in the detectionstate of the subject is greater than a predetermined amount, thephotographing target subject is determined to be moving. When the statein which both of the change amount of position or size of the subjectdetected by the subject detection unit 123 and the shake amount detectedby the shake detection unit 125 in the detection state of the subjectare less than the predetermined amount continues for a predeterminedtime or more, the photographing target subject is determined to bestopped. A time in which the detection state continues is assumed to bemeasured by a measurement timer.

Further, a method of calculating the distance between the camera and thesubject will be described. First, a focal distance serving as areference and a subject distance in regard to a subject size aremeasured in advance and measurement data is stored in the memory 118.The subject distance is calculated by performing calculation of “subjectdistance=(reference subject distance×focal distance×reference subjectsize)/(reference focal distance×detection size)” in regard to referencevalues stored in the memory 118. For example, when the reference valuesare the reference subject distance of 2 m, the reference focal distanceof 24 mm, and the reference subject size of 20 pix (pixels), a case inwhich a subject image with a detection size of 40 pix is detected with afocal distance of 120 mm is assumed. In this case, the subject distanceis calculated as “(2 mx120=×20 pix)/(24=×40 pix)=5 m.”

Next, a composition when multiple people are detected as subjects in ascreen will be described. When a plurality of people are detected assubjects in a screen in the composition of upper body 1400 or whole body1401 during the still image framing 1405, a process of changing thecomposition to multiple people 1403 is performed. The number of subjectsdetected at this time is stored in the memory 118. The composition ofmultiple people 1403 is a composition selected when all of the pluralityof subjects are determined to be a photographing scene of a group photofitted at a predetermined angle of view during the still image framing1405. In the composition state of multiple people 1403, a process ofdetermining whether the number of detected subjects is changed isperformed. When the number of detected subjects increases, a new subjectenters the screen, the number of subjects stored in the memory 118 isthus updated instantly, and the composition is changed. Conversely, whenthe number of detected subjects decreases, there is a possibility of atemporarily undetectable state, for example, as in a case in which onesubject is oriented horizontally. Therefore, when the composition isinstantly changed, there is a possibility of an undetectable subjecthaving fallen outside of the frame. Accordingly, when the number ofsubjects decreases, it is determined whether this state continues over apredetermined time or more. When this state continues for thepredetermined time or more, the number of subjects stored in the memory118 is updated and the composition is changed. Further, when the numberof subjects is one person, a process of returning the composition toupper body 1400 is performed.

When a composition in which all of the subjects are fitted at an angleof view is set during the moving image recording 1406 and a subjectother than the photographing target is moving, there is a concern of anerroneous operation of zoom occurring in response to this subject.Accordingly, when a plurality of people are detected as subjects duringthe moving image recording 1406, a process of setting only a mainsubject as a photographing target and maintaining upper body 1400 orwhole body 1401 as the composition is performed. When a face is detectednear the center of the screen over a predetermined time or more in thecomposition of upper body 1400 during the moving image recording 1406, aprocess of changing the composition to face 1402 is performed. Thecomposition of face 1402 is a composition selected when a scene in whichthe photographer is interested in the face of the subject is determinedduring the moving image recording 1406. In such a scene, the face of thesubject of interest can be photographed with a larger size by settingthe composition that approaches the face more than the upper body. Bychanging a zoom speed to a speed considerably lower or higher than anormal speed during a zoom-in operation by the change in thecomposition, a more effective video can be photographed. Further, when aface is detected near the periphery of a screen over a predeterminedtime or more in the composition of face 1402 or the photographer isdetermined to change framing, a process of returning the composition toupper body 1400 is performed. In the embodiment, a periphery region of ascreen is exemplified as a specific region, and a continuous time of astate in which the position of a face image of a subject detected by thesubject detection unit 123 is the periphery region is measured by ameasurement timer. As a method of determining the change of the framing,the photographer is determined to change the framing when the subjectdetection unit 123 does not detect the subject and the shake amountdetected by the shake detection unit 125 is greater than thepredetermined amount.

During the still image framing 1405, the photographer sometimescontinues to wait to take a picture in the same composition until thedirection or expression of the face of a person becomes a desired state.In this case, when the composition of approach to the face is set, thereis a possibility of the composition being a different composition fromthat intended by the photographer. Accordingly, even when the face isdetected near the center of the screen for a predetermined time or morein upper body 1400 during the still image framing 1405, a process ofmaintaining the composition of upper body 1400 is performed.

Next, a composition at the time of the change in a photographing stateby start/end of the moving image recording will be described. Thecompositions of upper body 1400 and whole body 1401 are compositionsselectable in both cases during the still image framing 1405 or duringthe moving image recording 1406. Accordingly, in these compositions, theoriginal composition is maintained even when the photographing state ischanged. On the other hand, the compositions of multiple people 1403 andface 1402 are compositions selected only in one case during the stillimage framing 1405 or during the moving image recording 1406.Accordingly, when the moving image recoding starts in the composition ofmultiple people 1403 during the still image framing 1405 or the movingimage recording ends in the composition of face 1402 during the movingimage recording 1406, the change to upper body 1400 which is a commoncomposition is considered. However, in this case, there is a concern ofan image being recorded in a zoom state in the beginning of a movingimage if the zoom operation starts along with the start or end of themoving image recording, or of the photographer feeling annoyed in thestill image framing. Accordingly, when the photographing state ischanged in such a state, a process of temporarily changing thecomposition to manual 1404 is performed. The composition of manual 1404is a composition in which the size of a main subject image firstdetected after the change of the composition is set to the referencesize. Accordingly, the zoom operation does not immediately start due tothe change of the photographing state and the photographer does not feelthe sense of discomfort. Even when a subject starts to move, the size ofa subject image can be maintained. Further, when a predetermined timehas passed in the composition of manual 1404, a process of returning thecomposition to upper body 1400 is performed.

FIG. 24 illustrates a transition diagram when the subject isauthenticated as a registered face. The same reference numerals aregiven to compositions common to FIG. 14 and the detailed descriptionthereof will be omitted. Compositions of selection candidates during thestill image framing 1405 are upper body 1400, whole body 1401, face1402, and multiple people 1403 and one composition is selected. FIG. 25illustrates an initial state and a composition change condition of thecomposition transition according to an age of the subject authenticatedin the face authentication process. In the embodiment, when the age ofthe authenticated subject is less than a predetermined age and a face isdetected over a predetermined time in the vicinity of the center of thescreen even during the still image framing 1405, a process of changingthe composition from upper body 1400 to face 1402 is performed.Specifically, when the subject is a 0-year-old baby and the photographerframes the face in the middle of the screen, the composition of approachto the face is selected. That is, when the subject is determined to be ababy who is relatively still (only 0 years is set as a compositiondetermination condition, unlike the age of a baby in the above-describedface ratio setting), the composition is changed so that a still imagecan be photographed with a close-up on the face. Since the predeterminedtime of the change of the composition from upper body 1400 to face 1402is set to be shorter than at other ages, the composition of face 1402 iseasily selected.

When the subject is a 1-year-old to 12-year-old child, photographingscenes of moving and playing are assumed to be many and the compositionof whole body 1401 is set to be easily selected. Therefore, when thesubject is a child, the composition of the initial setting is set towhole body 1401. A determination distance indicating the subjectdistance D1 or D2 is shorter than in the case of the determinationdistance for other ages so that upper body 1400 is changed to whole body1401 even when a distance to the subject is shorter.

As described above, when the composition setting is “auto” in S1211 ofFIG. 12, an appropriate composition is automatically determinedaccording to the photographing state, the number of subjects, a motionof the subject, or the like. It is possible to adjust the zoom to a moreappropriate angle of view by changing the composition selected accordingto the age of the subject. Data indicating the composition after thedetermination is stored in the memory 118 and the process proceeds toS1212.

In S1212, it is determined whether the composition determined in S1211is changed from the immediately previous composition (whether thecomposition selected in FIG. 14 transitions or the number of subjects ischanged in the state of multiple people 1403). When the composition ischanged, the process proceeds to S1213 to determine whether the selectedcomposition is the composition of manual 1404. When the selectedcomposition is the composition of manual 1404, the process proceeds toS1201 to perform a process of storing the size of the first detectedsubject image as the reference size in the memory 118. When the selectedcomposition is not the composition of manual 1404, the process proceedsto S1215 to calculate the reference size in the composition setting“auto.” When it is determined in S1212 that the composition is notchanged, the process proceeds to S1214 to determine whether the subjectimage moves in the screen. As a method of calculating a movement amountof the subject image in the screen, a reference size is set in S1215,and subsequently the central position of the face tracking frameimmediately after the zoom operation up to the angle of view for thereference size is stored in the memory 118. A distance from the positionstored in the memory 118 to the central position of the detected facetracking frame is calculated as a movement amount. The central positionof the face tracking frame may be added during each predetermined periodand the movement amount in the screen may be calculated. When thecalculated movement amount is equal to or greater than a predeterminedamount, the subject is determined to move. By normalizing thepredetermined amount with the size of the detected face tracking frame(setting a predetermined multiple of the size of the face trackingframe), a movement amount of an actual subject can be set to besubstantially the same irrespective of the size of the subject image inthe screen. When it is determined in S1214 that the subject moves in thescreen, the process proceeds to S1215 to calculate a reference size inthe composition setting “auto.” When it is determined in S1214 that thesubject does not move in the screen, the reference size setting processends without updating the reference size.

Hereinafter, the reference size calculation process in the compositionsetting “auto” will be described with reference to FIGS. 15A to 19. Whenthe composition setting is one of “face,” “upper body,” and “wholebody,” the reference size is calculated so that a face has a constantratio with respect to the screen size in S1210 of FIG. 12. That is, thesame reference size is calculated irrespective of the position of thesubject image on the screen. Therefore, when the composition is changedso that the subject image increases (is zoomed in on) in a case in whichthe subject is present in the periphery of the screen, there is apossibility of the subject falling outside of the frame. When thereference size is calculated with this method, it is necessary for thephotographer to frame the subject near the center of the screen.Accordingly, this method is not appropriate, for example, when thesubject is shifted from the center of the screen and is photographedalong with a background. When there are a plurality of subjects, it isnecessary to calculate the reference size with another method.Accordingly, when the composition setting is “auto,” the size of thesubject image is decided based on the composition determined in S1211and a process of calculating the reference size so that the subject isfitted in the screen irrespective of the position of the detectedsubject is performed.

FIGS. 15A to 15C are diagrams illustrating a relation between a screensize and the position of a subject closest to the periphery of thescreen among a plurality of subjects. FIGS. 15A and 15B illustratescreen examples of still image photographing and FIG. 15C illustrates ascreen example of moving image photographing and exemplifies a pluralityof people as subjects. FIG. 16 is a flowchart illustrating an overallprocess of calculating the reference size in the composition setting“auto.”

In S1600 of FIG. 16, a process of calculating a subject position ratioRh in the horizontal direction is first performed. The subject positionratio Rh in the horizontal direction is a ratio of a distance×2 up to ahorizontal position of a shoulder vicinity (hereinafter referred to as ashoulder position and denoted by Xsmax) of a subject closest to aperiphery from the center of a screen to a predetermined ratio withrespect to a screen size in the horizontal direction. The predeterminedratio is, for example, 80% or 90% of the screen size in the horizontaldirection and is changed according to the number of people arranged inthe horizontal direction in S1703 to be described below. In FIG. 15A, asubject closest to the periphery from the center of the screen in thehorizontal direction is a subject 1500 a among subjects 1500 a, 1501 a,and 1502 a. A process of calculating the subject position ratio Rh inthe horizontal direction will be described with reference to theflowchart of FIG. 17.

In step S1700, a shoulder position of the subject (denoted by Xs) iscalculated based on a central position or size of the face trackingframe of the subject. The center of the screen is assumed to be theorigin (0, 0), the central position of the face tracking frame isassumed to be (Xc, Yc), and the size of the face tracking frame isassumed to be S. When Ns is the number of face tracking frames locatedin a distance from the central position of the face tracking frame tothe shoulder position, the shoulder position Xs on the side of theperiphery of the screen can be calculated as “Xs=|Xc|+S×Ns”. FIG. 19illustrates a setting example of Ns. A value of Ns is changed accordingto the composition determined in S1211 of FIG. 12. For example, in thecomposition of face 1402, Ns=1 (in this case, a horizontal position ofthe vicinity of an ear rather than the shoulder position) is set. Inother cases (“upper body,” “whole body,” and “multiple people”), Ns=2 isset.

After the shoulder position Xs of the subject is calculated, the processproceeds to S1701. In S1701, the shoulder position Xs calculated inS1700 is stored as the maximum shoulder position Xsmax when the maximumvalue Xsmax of the shoulder position is not stored in the memory 118.When the maximum shoulder position Xsmax is stored in the memory 118,the shoulder position Xs calculated in S1700 is compared to the maximumshoulder position Xsmax. When the shoulder position Xs is greater thanthe maximum shoulder position Xsmax, a process of updating the maximumshoulder position Xsmax in the memory 118 to the shoulder position Xs isperformed. After the process of updating the maximum shoulder positionXsmax, the process proceeds to S1702. In S1702, it is determined whethera process of calculating the shoulder positions Xs of all the subjectsand updating the maximum shoulder positions Xsmax ends. When thecalculation of the shoulder position Xs and the updating of the maximumshoulder position Xsmax do not end, the process returns to S1700. Whenthe determination of the maximum shoulder positions Xsmax of all thesubjects ends and the updating of the maximum shoulder positions Xsmaxends, the process proceeds to S1703.

In S1703, a process of determining the number of people arranged assubjects in the horizontal direction (the number of people arranged inthe horizontal direction) is performed. This is because a composition inwhich the subjects are fully centered in a screen is set when the numberof subjects is larger as in a group photo, and a composition in which amargin remains in the periphery of the screen is set when the number ofsubjects is small. As a method of counting the number of people arrangedin the horizontal direction, when the face tracking frames overlap inthe vertical direction (the vertical direction of the screen), theoverlapping subjects are counted as one person in total. For example,when four people are detected as subjects in the screen and the facetracking frames of two people among the four people detected as subjectsoverlap in the vertical direction of the screen, the number of people iscounted as three. In S1703, the number of people arranged in thehorizontal direction is compared to a threshold value (for example, twopeople). When the number of people arranged in the horizontal directionis determined to be equal to or less than two, the process proceeds toS1704. When the number of people arranged in the horizontal direction isdetermined to be equal to or greater than three, the process proceeds toS1705. In S1704 and S1705, each subject position ratio Rh in thehorizontal direction is calculated. The subject position ratio Rh in thehorizontal direction in S1704 is calculated as a ratio of the maximumshoulder position Xsmaxx2 to 80% of the horizontal screen size. Thesubject position ratio Rh in the horizontal direction in S1705 iscalculated as a ratio of the maximum shoulder position Xsmaxx2 to 90% ofthe horizontal screen size. After S1704 or S1705, the process ofcalculating the subject position ratio Rh ends.

Next, a process of calculating the subject position ratio Rv in thevertical direction is performed in S1601 of FIG. 16. The subjectposition ratio Rv in the vertical direction is a ratio of a distance×2up to a vertical position of the head of the subject (hereinafterreferred to as a head position) or up to a vertical position of a body(hereinafter referred to as a body position) of the subject closest tothe periphery from the center of the screen to the predetermined ratioof the screen size in the vertical direction. The predetermined ratiois, for example, 90% of the screen size in the vertical direction.Hereinafter, the head position of the subject closest to the peripheryfrom the center of the screen is denoted by Yhmax and the body positionof the subject closest to the periphery from the center of the screen isdenoted by Ybmix. Of the subjects 1500 b, 1501 b, and 1502 b in FIG.15B, a subject whose head is closest to the periphery from the center ofthe screen in the vertical direction is the subject 1500 b, and asubject whose body is closest to the periphery from the center of thescreen is the subject 1501 b. A process of calculating the subjectposition ratio Rv in the vertical direction will be described withreference to the flowchart of FIG. 18.

In S1800 of FIG. 18, the head position (denoted by Yh) of the subject iscalculated based on the central position or size of the face trackingframe of the subject. The head position Yh can be calculated as“Yh=Yc+S×Nh”. Nh indicates the number of face tracking frames located ina distance from the central position of the face tracking frame to thehead position. A setting example of Nh is illustrated in FIG. 19. For avalue of Nh, Nh=1.5 is set irrespective of the composition determined inS1211 of FIG. 12. After the head position Yh of the subject iscalculated, the process proceeds to S1801.

In S1801, when a maximum value Yhmax of the head position is not storedin the memory 118, a process of storing the head position Yh calculatedin S1800 as a maximum head position Yhmax is performed. When the maximumhead position Yhmax is stored in the memory 118, the head position Yhcalculated in S1800 is compared to the maximum head position Yhmax. Whenthe head position Yh is greater than the maximum head position Yhmax, aprocess of updating the maximum head position Yhmax in the memory 118 tothe head position Yh is performed. After the updating process, theprocess proceeds to S1802.

Next, in S1802, a body position (denoted by Yb) of the subject iscalculated based on the central position or size of the face trackingframe of the subject. The body position Yb can be calculated as“Yb=Yc−S×Nb”. Nb indicates the number of face tracking frames located ina distance from the central position of the face tracking frame to thebody position. A setting example of Nb is illustrated in FIGS. 19 and26. FIG. 26 illustrates a setting example of Nb when the detectedsubject is a registered face. For example, in the case of theunauthenticated subject and the case of the composition of the face1402, Nb=1.5 is set. A value of Nb is changed according the compositiondetermined in S1211 of FIG. 12. For example, in the case of thecomposition of face 1402, Nb=1.5 is set. In the case of the compositionof upper body 1400, Nb=5 is set. In the case of the composition of wholebody 1401, Nb=10 is set. In the case of the composition of multiplepeople 1403, Nb=3.5 is set. In this setting, the center of the screen isset at a chest in face 1402, at a part below the waist in upper body1400, at a leg in whole body 1401, and at a position near a part abovethe waist in multiple people 1403. When the composition is one of upperbody 1400, whole body 1401, and multiple people 1403, the value of Nb ischanged in accordance with the age of the subject. In the embodiment,when the detected subject is not authenticated, the same Nb as the2-year-old to 12-year-old child is set. When the age of the subject is a0-year-old to 1-year-old baby, Nb is set to a value less than in theunauthenticated case. When the subject is an adult (in terms of bodyshape) of 13 years or older, Nb is set to a value greater than in theunauthenticated case. Accordingly, when the selected composition isupper body 1400, whole body 1401, multiple people 1403, it is possibleto reduce a case in which the subject is considerably larger orconsiderably smaller than the screen size. After the body position Yb ofthe subject is calculated, the process proceeds to S1803.

In S1803, when a minimum value Ybmin of the body position is not storedin the memory 118, a process of storing the body position Yb calculatedin S1802 as a minimum body position Ybmin is performed. When the minimumbody position Ybmin is stored in the memory 118, the body position Ybcalculated in S1802 is compared to the minimum body position Ybmin. Whenthe body position Yb is less than the minimum body position Ybmin, aprocess of updating the minimum body position Ybmin in the memory 118 asthe body position Yb is performed. After the updating process, theprocess proceeds to S1804.

In S1804, the head positions Yh and the body positions Yb of all thesubjects are calculated and it is determined whether a process ofupdating the maximum head position Yhmax and the minimum body positionYbmin ends. When the updating of the maximum head position Yhmax and theminimum body position Ybmin do not end, the process returns to S1800.When the determination of the maximum head positions Yhmax and theminimum body positions Ybmin of all the subjects ends and the updatingof the maximum head positions Yhmax and the minimum body positions Ybminends, the process proceeds to S1805. In S1805, the absolute value of themaximum head position Yhmax is compared to the absolute value of theminimum body position Ybmin and a process of determining which value islocated closer to the periphery of the screen is performed. When it isdetermined that the maximum head position Yhmax is located closer to theperiphery of the screen, the process proceeds to S1806. When it isdetermined that the minimum body position Ybmin is located closer to theperiphery of the screen, the process proceeds to S1807. In S1806 andS1807, each subject position ratio Rv in the vertical direction iscalculated. In S1806, the subject position ratio Rv in the verticaldirection is calculated as a ratio of the absolute value×2 of themaximum head position Yhmax to 90% of the vertical screen size. InS1807, the subject position ratio Rv in the vertical direction iscalculated as a ratio of the absolute value×2 of the minimum bodyposition Ybmin to 90% of the vertical screen size. After S1806 or S1807,the process of calculating the subject position ratio Rv in the verticaldirection ends.

Subsequently, in S1602 of FIG. 16, the subject position ratio Rh in thehorizontal direction is compared to the subject position ratio Rv in thevertical direction. Accordingly, the position at which the ratio is thelargest among the ratios of the distances from the center of the screento the respective positions of the subject in regard to thepredetermined ratio of the screen size in each direction, that is, theposition closest to the periphery of the screen, can be determined. Therespective positions are the maximum shoulder position Xsmax, themaximum head position Yhmax, and the minimum body position Ybmin. Whenthe subject position ratio Rh in the horizontal direction is determinedto be greater than the subject position ratio Rv in the verticaldirection, the process proceeds to S1603. When the subject positionratio Rh in the horizontal direction is determined to be equal to orless than the subject position ratio Rv in the vertical direction, theprocess proceeds to S1604. In S1603 and S1604, a process of calculatingthe reference size is performed based on the detection size of the mainsubject and the ratio of the subject position closest to the peripheryof the screen determined in S1602. The reference size is calculated as“detection size/subject position ratio Rh in horizontal direction” inS1603 and is calculated as “detection size/subject position ratio Rv invertical direction” in S1604. That is, in S1603, the zoom operation isperformed so that the main subject image has the reference size obtainedby multiplying the size of the main subject image by “1/subject positionratio Rh in horizontal direction”. In S1604, the zoom operation isperformed so that the main subject image has the reference size obtainedby multiplying the size of the main subject image by “1/subject positionratio Rv in vertical direction”. In this way, the subject can be set atthe angle of view based on the composition determined in S1211 withoutthe subject falling outside of the screen.

When the composition setting is “auto” in S1215 of FIG. 12, the subjectposition closest to the screen periphery in the horizontal direction andthe vertical direction is determined based on the composition determinedin S1211 and the reference size is calculated so that this subjectposition is fitted in the screen. Accordingly, regardless of a positionon the screen at which the subject image is located, the subject can befitted at an appropriate angle of view without falling outside of theframe. Whether the number of subjects is singular or plural, thereference size can be calculated through the same process. In theembodiment, the direction of the screen size by which the reference sizeis calculated has been decided on the presupposition that the personserving as a subject is standing. The invention is not limited thereto.As long as the subject detection unit 123 can determine the direction ofthe face, the direction of the screen size may be decided based on thevertical direction of the face on the screen. Further, the compositiondetermination or the calculation example of the reference size when thesubject is a person has been described, but the invention can also beapplied when the subject is an object. However, in this case,compositions to be selected are assumed to be “large,” “medium,”“small,” and “multiple” instead of “face,” “upper body,” “whole body,”and “multiple people.” According to the compositions, the number ofobject tracking frames (corresponding to Ns, Nh, and Nb in the case of aperson) when the subject position is calculated may be set. When thesubject can be authenticated through the object authentication process,the composition may be changed in accordance with the authenticatedsubject. For example, as illustrated in FIG. 27, when a subject is amoving object such as an airplane, an electric train, or a bird, amotion of the subject is determined according to the above-describedmethod of determining whether the subject is moving or still. When thesubject is determined to be moving, the composition is set to be“small.” When the subject is a moving object and the object isdetermined to be still, the composition is set to “medium.” When thesubject is a stationary object such as the moon, the composition is setto “large.” In this way, even when the subject is an object other than aperson, the composition may be changed according to an authenticationresult.

When the reference size setting process illustrated in FIG. 12 ends, theprocess proceeds to S704 of FIG. 7. In S704, the AZ control unit 122performs automatic zoom control based on the subject informationdetected by the subject detection unit 123 and the reference size storedin the memory 118. The automatic zoom control will be described belowwith reference to FIG. 20. After the automatic zoom control ends, theprocess proceeds to S705. Then, the AZ control unit 122 determineswhether the photographer gives an instruction to search for a subject.That is, the AZ control unit 122 determines whether the photographerpresses the subject search button of the manipulation unit 117. When thesubject search button is pressed, the process returns to S701 and thesubject searching process is performed. Conversely, when the subjectsearch button is not pressed, the process proceeds to S706. In S706, itis determined whether the automatic zoom target subject is changedthrough manipulation of the touch panel, a subject designation switch,or the like of the manipulation unit 117. When the subject is changed,the process returns to S702 to perform the subject designation process.Conversely, when the subject is not changed, the process proceeds toS707.

In S707, it is determined whether the composition setting is changedthrough manipulation of the left and right buttons of the manipulationunit 117 from the photographing screen. When the composition setting ischanged, data of the composition setting stored in the memory 118 isupdated, and subsequently the process returns to S703 to perform thereference size setting process. Conversely, when the composition settingis not changed, the process proceeds to S708. In S708, it is determinedwhether the automatic zoom manipulation switch of the manipulation unit117 is pressed. When the automatic zoom manipulation switch isdetermined to be pressed, the automatic zoom function ends. Conversely,when the automatic zoom manipulation switch is determined not to bepressed, the process proceeds to step S709. In S709, a process ofdetermining whether the composition setting stored in the memory 118 is“auto” is performed. When the composition setting is “auto,” the processreturns to S703 to perform the process of setting the reference sizeincluding the automatic composition determination. Conversely, when thecomposition setting is setting other than “auto,” the process returns toS704 to continue the automatic zoom control.

Next, the automatic zoom control process of S704 will be described withreference to the flowchart of FIG. 20. In S2000, the AZ control unit 122first determines whether the subject detection unit 123 detects thesubject. When the subject is not detected in S2000, the automatic zoomcontrol ends. When the subject is detected in S2000, the processproceeds to S2001. The determination of steps from S2001 to S2003 is anautomatic zoom start determination process for control of preventing asubject from falling outside of the frame. That is, this process isperformed to prevent a subject from falling outside of the screen, asdescribed in FIGS. 3 to 4C. In S2001, the AZ control unit 122 determineswhether a subject tracking frame related to a tracking target subjectenters a ZO region. This ZO region corresponds to the ZO regiondescribed in the left figure of FIG. 3 and FIGS. 4A to 4C. When thesubject tracking frame enters the ZO region in S2001, that is, there isa high possibility of the subject falling outside of the frame, theprocess proceeds to S2004 to start a zoom-out operation. Here, thezoom-out operation corresponds to a zoom-out operation for the controlof preventing the subject from falling outside of the frame. After thezoom-out operation, the automatic zoom control ends.

Conversely, when the subject tracking frame does not enter the ZO regionin S2001, that is, when the camera captures the subject near the centerof the screen, the process proceeds to S2002. In S2002, it is determinedwhether an immediately previous zoom operation is a zoom-out operationperformed because the subject tracking frame entered the ZO region inS2001, that is, whether the immediately previous zoom operation is thezoom-out operation for the control of preventing the subject fromfalling outside of the frame. When it is determined in S2002 that theoperation is performed after the zoom-out operation for the control ofpreventing the subject from falling outside of the frame, the processproceeds to S2003. Conversely, when it is determined in S2002 that theoperation is not performed after the zoom-out operation for the controlof preventing the subject from falling outside of the frame, the processproceeds to S2006.

In S2003, the AZ control unit 122 determines whether the subjecttracking frame related to the tracking target subject is fitted(included) in a ZI region. Here, this ZI region corresponds to the ZIregion described in the right figure of FIG. 3B and FIGS. 4A to 4C. Whenit is determined in S2003 that the subject tracking frame is not fittedin the ZI region, the automatic zoom control process ends. Conversely,when it is determined that the subject tracking frame is fitted in theZI region, the process proceeds to S2005. That is, when the cameracaptures the subjects so that the subject size is within an angle ofview near the center of the screen and at a zoom return position, azoom-in operation starts in S2005. Here, the zoom-in operationcorresponds to the zoom-in operation for the control of preventing thesubject from falling outside of the frame. After the zoom-in operation,the automatic zoom control ends.

In the embodiment, to cause the control of preventing the subject fromfalling outside of the frame and the size maintenance control in theautomatic zoom to be compatible, the subject is captured near the centerof the screen for the control of preventing the subject from fallingoutside of the frame and the size maintenance control can be performed.Therefore, after the zoom-out operation for the control of preventingthe subject from falling outside of the frame, an automatic zoom process(S2006 to S2010 below) of constantly maintaining a subject size to bedescribed below (controlling size maintenance) is not performed. Inother words, when the control of preventing the subject from fallingoutside of the frame is performed, the size maintenance control is notperformed until the zoom-in operation for the control of preventing thesubject from falling outside of the frame is completed.

Next, processes of S2006 to S2010 will be described. When it isdetermined in S2002 that the operation is not performed after thezoom-out operation for the control of preventing the subject fromfalling outside of the frame, the process proceeds to S2006. The AZcontrol unit 122 compares the detection size of the subject to a sizeobtained by multiplying the reference size by a predetermined ratio(denoted by N1 where N1>1). The size of the subject indicated by theinformation regarding the reference subject set in S1202 or S1210 andS1215 of FIG. 12 is compared to the size of the subject detected inS2000. When the size of the subject detected in S2000 is greater than N1times the size of the reference subject, that is, a ratio of the subjectimage to the screen exceeds a predetermined value, the process proceedsto S2009. In S2009, the AZ control unit 122 starts a zoom-out operation.Here, this zoom-out operation corresponds to the zoom-out operation forthe size maintenance control. After the zoom-out operation, theautomatic zoom control ends.

Conversely, when the size of the subject detected in S2000 is equal toor less than N1 times the size of the reference subject in S2006, theprocess proceeds to S2007. In S2007, the AZ control unit 122 comparesthe size of the subject of the information regarding the referencesubject to the size of the subject detected in S2000. When the size ofthe detected subject is less than the size of the reference subject at apredetermined ratio (denoted by N2 where N2<1) (less than N2 times),that is, a ratio of the subject image to the screen is less than apredetermined value, the process proceeds to S2008. Conversely, when thesize of the detected subject is equal to or greater than N2 times thesize of the reference subject, the automatic zoom control ends.

In S2008, the AZ control unit 122 determines whether the subjecttracking frame related to the tracking target subject is fitted(included) in a ZI region. This is because the subject is prevented fromfalling outside of the frame by the zoom-in operation when the subjectis in the periphery of the screen. Here, this ZI region corresponds tothe ZI region described in the center figure of FIG. 5B. When it isdetermined in S2008 that the subject tracking frame is not fitted in theZI region, the automatic zoom control process ends.

Conversely, when it is determined in S2008 that the subject trackingframe is fitted in the ZI region, the process proceeds to S2010. InS2010, the AZ control unit 122 starts a zoom-in operation. Here, thiszoom-in operation corresponds to the zoom-in operation for the sizemaintenance control. In this way, in the embodiment, to prevent thesubject from falling outside of the frame even in the zoom-in operationfor the size maintenance control, the subject image is fitted in the ZIregion and then the zoom-in operation starts. After the zoom-inoperation, the automatic zoom control ends.

Next, a zoom operation will be described with reference to FIG. 21. FIG.21 is a flowchart for describing the zoom-out operation or the zoom-inoperation in S2004, S2005, S2009, and S2010 of FIG. 20. First, in S2100,the AZ control unit 122 acquires a zoom change amount (a change amountof zoom magnification) from the memory 118. In the case of the zoom-outoperation for the control of preventing the subject from falling outsideof the frame, a zoom-out change amount is set according to theinformation regarding the detected subject.

Specifically, in the zoom-out operation for the control of preventingthe subject from falling outside of the frame (S2004 of FIG. 20), thezoom-out change amount is set to be smaller as the size of the subjectis smaller. Accordingly, it is possible to prevent the subject frombecoming too small due to the zoom-out operation and not being detected.In consideration of a detectable minimum size of the subject, thezoom-out operation may not be performed when the size of the subject isless than a predetermined size. In a zoom-in operation for the controlof preventing the subject from falling outside of the frame, a zoommagnification before start of the zoom-out operation is stored in thememory 118. A zoom-in change amount is set so that the zoommagnification is the same as the zoom magnification before the start ofthe zoom-out operation.

In the zoom-out operation for the size maintenance control (S2009 ofFIG. 20), a zoom-out change amount (1/N1 times) corresponding to apredetermined N1 times used for the determination of S2006 is set.Accordingly, even when the subject is undetectable, a minimum zoom-outoperation can be performed until the size of the subject becomes thesize of the reference subject. Similarly, in the zoom-in operation forthe size maintenance control (S2010 of FIG. 20), a zoom-in change amount(1/N2 times) corresponding to a predetermined N2 times used for thedetermination of S2007 is also set.

In S2101, the AZ control unit 122 sets the zoom change amount acquiredin S2100 in the CZ control unit 119 or the electronic zoom control unit120 and gives an instruction to perform a magnification process. Next,in S2102, the AZ control unit 122 determines which zoom operation isbeing performed between the zoom operations for the control ofpreventing the subject from falling outside of the frame and for thesize maintenance control. When the zoom operation at the current time isthe zoom operation for the control of preventing the subject fromfalling outside of the frame (S2004 and S2005 of FIG. 20), the processproceeds to S2105. Conversely, when the zoom operation at the currenttime is the zoom operation for the size maintenance control (S2009 andS2010 of FIG. 20), the process proceeds to S2103.

In S2103, the AZ control unit 122 determines whether the subjectdetection unit 123 detects the subject. When the subject is detected,the process proceeds to S2104. When the subject is not detected, theprocess proceeds to S2105. In S2104, the size of the subject indicatedby the information regarding the reference subject is compared to thesize of the subject detected in S2103. When the size of the subjectdetected in S2103 and the size of the reference subject are determinednot to be within a predetermined ratio range (within a predeterminedchange amount) as the result of the comparison, the process proceeds toS2102 again and the determination of the zoom operation continues. Whenthe size of the subject detected in S2103 and the size of the referencesubject are within the predetermined ratio range through the zoomoperation, the process proceeds to S2106. In S2106, the AZ control unit122 stops the zoom operation, and then ends the zoom operation process.

When it is determined in S2102 that the zoom operation is the zoomoperation for the control of preventing the subject from falling outsideof the frame or the subject is not detected during the zoom operationfor the size maintenance control in S2103, the process proceeds toS2105. In S2105, based on the zoom change amount acquired in S2100, theAZ control unit 122 determines whether a magnification processcorresponding to the zoom change amount of a predetermined amountaccording to each zoom operation is performed. When the magnificationprocess of the predetermined zoom change amount is not performed, theprocess returns to S2102 again to continue the process. Conversely, whenthe magnification process of the predetermined zoom change amount isperformed, the process proceeds to S2106 and the AZ control unit 122stops the zoom during the operation and ends the zoom operation process.

In the embodiment, a photographing scene is determined in accordancewith a motion of a subject, the number of subjects, a detection time ofa subject, a distance from a camera to a subject, or the like. Theappropriate composition is automatically selected by the cameraaccording to the determined scene. A process of setting the referencesize of the subject image is performed based on the composition selectedthrough the composition selection process and a zoom operation isperformed so that an image of the detected subject has the referencesize. According to the embodiment, the automatic zoom can be performedso that the appropriate composition is selected according to thephotographing scene determined based on the detection information of thecamera without selection of the composition by the photographer.

In the embodiment, the more appropriate composition is automaticallyselected by the camera based on the attribute information of the subjectin accordance with the authentication result of the subject. The processof setting the reference size of the subject image is performed based onthe composition selected through the composition selection process andthe zoom operation is performed so that the image of the detected imagehas the reference size. In particular, by changing the face ratioaccording to age in the calculation of the reference size of a person,it is possible to perform the zoom operation so that the moreappropriate size is set. In the embodiment, the case in which theselection of the composition or the face ratio is changed in accordancewith the age of a person has been exemplified, but the invention is notlimited thereto. For example, a configuration in which the initialcomposition or the face ratio is registered for each individual (name)may be realized. The example in which ages are classified into baby (0years old or 0 to 1 year old), child (12 years old or less), and adult(13 year old or more) has been described. The ages may be set in moredetail or the ages may be changed in accordance with sex. In theembodiment, the automatic zoom control can be performed with thecomposition appropriate for the attribute of the subject determinedbased on the authentication information of the subject without selectingthe composition of the subject.

Second Embodiment

Next, a second embodiment of the invention will be described withreference to FIG. 22. FIG. 22 illustrates an example of the transitiondiagram of compositions different from FIG. 14 in the automaticcomposition determination process S1211 of the composition setting“auto” in FIG. 12. The same reference numerals are given to compositionscommon to FIG. 14 and the detailed description thereof will be omitted.

In the first embodiment, the example in which the photographing scene isdetermined when a person different from the photographer is photographedas a subject has been described. The mode at that time is referred to asa normal mode for convenience. A camera is directed to a photographerhimself or herself to photograph one person or multiple people includingthe photographer himself or herself as subjects in some cases. A modeappropriate for such photographing is referred to as aself-photographing mode. In the self-photographing mode, by adjusting anangle of view appropriate for the self-photographing through automaticzoom control and applying a background blurring effect, a skinbeautification filtering effect, or the like, it is possible to performeffective portrait photographing.

In a composition matching manipulation, it is necessary to set a subjectimage to an appropriate size according to whether the subject is stoppedor moving at the time of the normal mode. On the other hand, in theself-photographing mode, it is necessary to change an appropriate sizeaccording to whether a photographer is holding a camera or has put itdown even when the subject is stopped. For example, when thephotographer is holding the camera to perform photographing, thephotographing is performed so that a face is fitted to be large at theangle of view. When the photographer has put the camera down to performphotographing, the photographing is performed so that a whole body isfitted at an angle of view. In this way, an appropriate compositiondiffers between the photographing of the normal mode and theself-photographing mode.

Accordingly, in the embodiment, a process of changing the selectedcomposition between a normal mode 2200 and a self-photographing mode2201 in FIG. 22 is performed. That is, the normal mode is exemplified asthe first mode and the self-photographing mode is exemplified as thesecond mode. One composition is selected from upper body 1400, wholebody 1401, and multiple people 1403 which are compositions of theselection candidates belonging to the first composition group in thenormal mode 2200. One composition is selected from self-photographingface 2202 and self-photographing whole body 2203 which are compositionsof the selection candidates belonging to the second composition group inthe self-photographing mode 2201.

Here, a determination condition of the photographing scene andtransition of each composition will be described. When the automaticcomposition determination process S1211 of the composition setting“auto” starts, upper body 1400 is set as the composition of the initialstate. When the mode is the normal mode 2200, the transition is the sameas that during the still image framing 1405 of FIG. 14. Therefore, thedescription thereof will be omitted.

When the mode is changed from the normal mode 2200 to theself-photographing mode 2201, the shake detection unit 125 determineswhether the camera is held in the photographer's hand based on adeflection amount applied to the gyro sensor or the like. When the modeis changed to the self-photographing mode 2201 and when the deflectionamount is equal to or greater than the predetermined amount and thus thecamera is determined to be held in the photographer's hand, thecomposition is changed to self-photographing face 2202. The compositionof self-photographing face 2202 is a composition of a close-up on thephotographer's face with the camera held in the photographer's hand toperform photographing. Even when multiple people including thephotographer are photographed, zoom is adjusted so that the faces of allthe subjects are fitted at the angle of view to be as large as possible.

In the case of the self-photographing mode 2201, the same composition isselected when the photographer is the only subject and when multiplepeople are the subjects. That is, the number of subject positionestimation faces illustrated in FIG. 19 is set to the same value. Thenumber of subject position estimation faces is the number of trackingframes (Ns, Nh, Nb) from the central position of the face tracking framefor estimating a shoulder position Xs, a head position Yh, and a bodyposition Yb of a subject in the reference size calculation process S1215of the composition setting “auto” in FIG. 12, as described above. Inparticular, by setting the number of tracking frames Ns up to theshoulder position and the number of tracking frames Nb up to the bodyposition to be different according to the selected composition, the samereference size is calculated so that this composition is set. In thefirst embodiment, when a subject of one person is photographed with thecomposition of “face,” the number of tracking frames from the centralposition (Xc, Yc) of the face tracking frame is the shoulder positionNs=1 and the body position Nb=1.5 and the reference size in which thevicinity of the ear or chest of the subject is set as a range fitted atthe angle of view is calculated. When the plurality of subjects arephotographed with the composition of “multiple people,” the number oftracking frames from the central position (Xc, Yc) of the face trackingframe is the shoulder position Ns=2 and the body position Nb=3.5 and thereference size in which the vicinities of the shoulders or waists of allthe subjects are set as a range fitted at the angle of view iscalculated. On the other hand, in the case of self-photographing face2202, the shoulder position Ns=1 and the body position Nb=1.5 which arethe same number of tracking frames as “face” are set irrespective of thenumber of subjects. Accordingly, the reference size in which thevicinities of the ears or chests of all the subjects are set as a rangefitted at the angle of view is calculated, and the zoom is adjusted tothe angle of view appropriate for the self-photographing when thephotographer is holding the camera.

When the composition is changed to self-photographing face 2202, thenumber of detected subjects is stored in the memory 118. In thecomposition state of self-photographing face 2202, a process ofdetermining whether the number of detected subjects is changed isperformed. When the number of detected subjects increases, the number ofsubjects stored in the memory 118 is updated instantly and the referencesize is changed. When the number of subjects decreases while this statecontinues for a predetermined time or more, the number of subjectsstored in the memory 118 is updated and the composition is changed. Thisis because when one subject is oriented horizontally and is temporarilyundetectable as in the case of multiple people 1403, the subject isprevented from falling outside of a frame by changing the compositioninstantly.

When the mode is changed from the normal mode 2200 to theself-photographing mode 2201, the shake detection unit 125 determineswhether the camera is put on (fixed to) a desk or the like based on thedeflection amount applied to a gyro sensor or the like. When the mode ischanged to the self-photographing mode 2201 and the deflection amount isless than the predetermined amount and thus the camera is determined tobe put down, the composition is changed to self-photographing whole body2203. The composition of self-photographing whole body 2203 is acomposition when the camera is put down and the photographer's wholebody is photographed. When multiple people including the photographerare photographed, the zoom is adjusted so that the whole bodies of allthe subjects are fitted at the angle of view. In the case ofself-photographing whole body 2203, the shoulder position Ns=2 and thebody position Nb=10 which are the same number of tracking frames as“whole body” are set irrespective of the number of subjects.Accordingly, the reference size in which the shoulders and legs of allthe subjects are set as a range fitted at the angle of view iscalculated and the zoom is adjusted to the angle of view appropriate forthe self-photographing in the state in which the camera is put down.When the composition is changed to self-photographing whole body 2203,the number of detected subjects is stored in the memory 118. When thenumber of subjects is changed as in self-photographing whole body 2203or self-photographing face 2202, the number of subjects stored in thememory 118 is updated and the reference size is changed.

In the embodiment, the photographing scene determined according to theset mode is changed and the composition more appropriate for each modeis automatically selected by the camera. In the embodiment, the case inwhich the appropriate composition is selected from a plurality ofcomposition selection candidates in one mode has been exemplified, butonly one composition is a selection candidate according to the mode.Further, the cases of the normal mode and the self-photographing modehave been exemplified as the modes, but the invention is not limited tothese modes. Besides the normal mode and the self-photographing mode,for example, an embodiment in which the composition of “whole body” of acompetitor is selected in a sports mode or an embodiment in which thecomposition of “face” of a baby is selected in a baby mode may berealized.

In the embodiment, the automatic zoom control can be performed so thatan appropriate composition is selected according to the photographingscene determined based on the set mode and the detection information ofthe camera without selection of the composition by the photographer.

Other Embodiments

The invention can also be realized as a process in which a programrealizing one or more functions of the above-described embodiments issupplied to a system or a device via a network or a storage medium, andone or more processors in a computer of the system or the device readsand executes the program. The invention can also be realized by acircuit (for example, an ASIC) realizing one or more functions.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described Embodiments and/or that includes one or morecircuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedEmbodiments, and by a method performed by the computer of the system orapparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described Embodiments and/or controlling theone or more circuits to perform the functions of one or more of theabove-described Embodiments. The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)?), a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-260109, filed Dec. 24 2014, Japanese Patent Application No.2015-202330, filed Oct. 13 2015, and Japanese Patent Application No.2015-202331, filed Oct. 13 2015, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. A zoom control device comprising: a subject detection unit configured to detect a subject image from an image; a composition selection unit configured to automatically select one composition among a plurality of compositions based on information regarding the subject image detected by the subject detection unit; a setting unit configured to set a reference size of the subject image used to control a zoom magnification based on the composition selected by the composition selection unit and a size and a position of the subject image detected by the subject detection unit; and a control unit configured to control the zoom magnification based on the reference size set by the setting unit and the size of the subject image detected sequentially by the subject detection unit.
 2. The zoom control device according to claim 1, wherein the composition selection unit changes the composition to be selected according to the number of subject images detected by the subject detection unit.
 3. The zoom control device according to claim 1, wherein the composition selection unit changes the composition to be selected according to a motion of the subject image determined based on the size of the subject image or the position of the subject image in the image.
 4. The zoom control device according to claim 1, further comprising: a detection unit configured to detect a motion of the zoom control device, wherein the composition selection unit changes the composition to be selected according to the motion of the zoom control device.
 5. The zoom control device according to claim 1, wherein the composition selection unit changes the composition to be selected according to a change in a distance between the zoom control device and a subject determined based on the size of the subject image detected by the subject detection unit and information regarding the zoom magnification controlled by the control unit.
 6. The zoom control device according to claim 1, wherein a plurality of photographing modes are provided, and wherein the composition selection unit changes a selection candidate composition according to the set photographing mode.
 7. The zoom control device according to claim 6, wherein the photographing modes include a first mode used to photograph a subject different from a photographer and a second mode used to photograph a subject including the photographer.
 8. The zoom control device according to claim 1, further comprising: a registration unit configured to associate image information regarding a subject with attribute information regarding the subject and register the image information and the attribute information; an authentication unit configured to authenticate the subject from a correlation value between a feature amount of the subject image detected by the subject detection unit and a feature amount of the subject image obtained from the image information of the subject registered in the registration unit
 9. The zoom control device according to claim 8, wherein the composition selection unit changes a condition for changing the composition according to the attribute information regarding the subject authenticated by the authentication unit.
 10. The zoom control device according to claim 8, wherein the setting unit changes the reference size according to the attribute information of the subject authenticated by the authentication unit.
 11. The zoom control device according to claim 1, wherein the control unit controls the zoom magnification so that the size of the subject image detected by the subject detection unit is within a range of a predetermined ratio with respect to the reference size.
 12. A zoom control device comprising: a subject detection unit configured to detect a subject image from an image; a registration unit configured to associate image information regarding a subject with attribute information regarding the subject and register the image information and the attribute information; an authentication unit configured to authenticate the subject from a correlation value between a feature amount of the subject image detected by the subject detection unit and a feature amount of the subject image obtained from the image information of the subject registered in the registration unit; a composition selection unit configured to automatically select one composition among a plurality of composition candidates; a setting unit configured to set a reference size of the subject image used to control a zoom magnification based on the composition selected by the composition selection unit and a size and a position of the subject image detected by the subject detection unit; and a control unit configured to control the zoom magnification based on the reference size set by the setting unit and the size of the subject image sequentially detected by the subject detection unit, wherein the composition selection unit changes the composition candidate selectable according to the attribute information regarding the subject authenticated by the authentication unit.
 13. The zoom control device according to claim 12, wherein the composition selection unit changes an initial composition selected when the subject is authenticated, according to the attribute information regarding the subject authenticated by the authentication unit.
 14. The zoom control device according to claim 12, wherein, in regard to the composition selected by the composition selection unit, the setting unit changes the reference size according to the attribute information regarding the subject authenticated by the authentication unit.
 15. The zoom control device according to claim 13, wherein the composition selection unit changes the composition from the initial composition according to information regarding the subject image detected by the subject detection unit.
 16. The zoom control device according to claim 15, wherein the composition selection unit changes a condition for changing the composition according to the attribute information of the subject authenticated by the authentication unit.
 17. The zoom control device according to claim 12, wherein, when the subject detected by the subject detection unit is a face of a person, the registration unit registers one or more pieces of information among an age, a birth state, a sex, an initial composition of the person as the attribute information regarding the subject.
 18. The zoom control device according to claim 12, wherein, when the subject detected by the subject detection unit is an object, the registration unit registers one or more pieces of information among a kind of the object, determination information of a moving object or a stationary object, and an initial composition as the attribute information regarding the subject.
 19. The zoom control device according to claim 12, wherein the control unit controls the zoom magnification so that the size of the subject image detected by the subject detection unit is within a range of a predetermined ratio with respect to the reference size.
 20. The zoom control device according to claim 12, wherein the plurality of compositions differ in a ratio of the size of the subject image to a screen size.
 21. A control method performed in a zoom control device, the method comprising: acquiring information regarding a subject image detected from an image; automatically selecting one composition among a plurality of compositions based on the information regarding the subject image; setting a reference size of the subject image used to control a zoom magnification based on the composition selected in automatically selecting and a size and a position of the subject image indicated by the information regarding the subject image; and controlling the zoom magnification based on the reference size set in the setting and the size of the subject image indicated by the information regarding the subject image sequentially acquired in the acquiring.
 22. A control method performed in a zoom control device, the method comprising: associating image information regarding a subject with attribute information regarding the subject and registering the image information and the attribute information; acquiring information regarding a subject image detected from an image; authenticating the subject from a correlation value between a feature amount of the subject image based on the detection information regarding the subject image and a feature amount of the subject image obtained from the registered image information of the subject; automatically selecting one composition among a plurality of composition candidates; setting a reference size of the subject image used to control a zoom magnification based on the composition selected in the automatically selecting and a size and a position of the subject image indicated by the information regarding the subject image; and controlling the zoom magnification based on the reference size set in the setting and the size of the subject image indicated by the information regarding the subject image sequentially acquired in the acquiring, wherein, in the automatically selecting, the composition candidate selectable according to the attribute information regarding the subject authenticated in the authenticating is changed.
 23. A non-transitory recording medium storing a control program of a zoom control device causing a computer to perform each step of a control method performed by the zoom control device, the method comprising: acquiring information regarding a subject image detected from an image; automatically selecting one composition among a plurality of compositions based on the information regarding the subject image; setting a reference size of the subject image used to control a zoom magnification based on the composition selected in automatically selecting and a size and a position of the subject image indicated by the information regarding the subject image; and controlling the zoom magnification based on the reference size set in the setting and the size of the subject image indicated by the information regarding the subject image sequentially acquired in the acquiring.
 24. A non-transitory recording medium storing a control program of a zoom control device causing a computer to perform each step of a control method of the zoom control device, the method comprising: associating image information regarding a subject with attribute information regarding the subject and registering the image information and the attribute information; acquiring information regarding a subject image detected from an image; authenticating the subject from a correlation value between a feature amount of the subject image based on the detection information regarding the subject image and a feature amount of the subject image obtained from the registered image information of the subject; automatically selecting one composition among a plurality of composition candidates; setting a reference size of the subject image used to control a zoom magnification based on the composition selected in the automatically selecting and a size and a position of the subject image indicated by the information regarding the subject image; and controlling the zoom magnification based on the reference size set in the setting and the size of the subject image indicated by the information regarding the subject image sequentially acquired in the acquiring, wherein, in the automatically selecting, the composition candidate selectable according to the attribute information regarding the subject authenticated in the authenticating is changed. 